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Dynamic  Resource  Allocation  and  Adaptability  in  Teamwork 

Steve  W.  J.  Kozlowski,  Richard  P.  DeShon, 

Guihyun  Park,  Paul  Curran,  Goran  Kuljanin,  and  Brady  Firth 

Background,  Research  Objectives,  and  Approach 


Problem  Background 

Team  performance  and  adaptability.  A  team  is  a  set  of  two  or  more  people  who  interact,  dynamically, 
interdependently,  and  adaptively  toward  a  common  and  valued  goal,  each  having  specific  roles  or  functions  to 
perform,  and  a  limited  life-span  of  membership  (Salas,  Dickinson,  Converse,  &  Tannenbaum,  1992).  We 
assume  that  all  cognition  originates  within  the  individual.  Therefore,  to  understand  adaptive  team  processes  it 
is  important  to  understand  the  ways  in  which  being  a  team  member  affects  individual  cognitive  processes.  We 
also  assume  that  unique  collective  constructs  and  processes  emerge  at  the  team  level  from  the  dynamic 
interaction  of  team  members  that  do  not  exist  at  the  individual  level  of  analysis,  despite  arising  from  individual 
cognition  (Kozlowski  &  Bell,  2003;  Kozlowski  &  Klein,  2000).  Finally,  we  focus  on  interdependent  tasks  in 
which  team  performance  is  a  weighted  function  of  actions  taken  by  team  members  to  accomplish  both 
individual  and  team  goals  (Shiflett,  1979;  Steiner,  1972). 

Many  critical  command,  control,  and  communication  activities  are  accomplished  by  individuals  operating  in 
teams  and  interacting  via  complex,  computer-mediated  systems.  These  dynamic  decision  making  task 
environments  place  high  demands  on  operator  skills  and  capabilities.  Such  tasks  are  dynamic,  ambiguous, 
and  emergent.  They  necessitate  rapid  situation  assessment,  prioritization,  and  strategy  implementation.  And, 
they  require  that  individuals  and  teams  adapt  their  performance  as  the  situation  shifts  and  unfolds — often 
unexpectedly.  A  key  factor  underlying  this  dynamic  process  of  assessment,  prioritization,  and  adaptability 
rests  on  the  capability  of  individuals  to  appropriately  allocate  limited  cognitive  and  behavioral  resources  to 
accomplish  multiple  goals  that  contribute  to  effective  individual  and  team  performance,  and  to  shift  their 
resource  allocations  to  meet  dynamic  task  demands.  This  task  structure  is  consistent  with  Steiner’s  (1972) 
most  general  type  of  team  task  (i.e.,  a  discretionary  task),  in  which  team  members  have  latitude  in  terms  of 
how  and  how  much  of  their  personal  resources  they  allocate  to  accomplish  team  performance  (Shiflett,  1979). 
Such  teams  require  each  member  to  assume  individual  goals,  but  to  also  coordinate  effort  and  provide 
assistance  to  other  team  members  to  meet  distinct  team  objectives.  That  is,  it  is  the  responsibility  of  individual 
team  members  to  make  resource  allocation  decisions  that  contribute  to  the  team,  such  as  choosing  to 
coordinate  collective  effort,  back-up  a  teammate,  or  aid  a  teammate  in  resolving  a  problem.  The  degree  to 
which  members  allocate  attention  and  effort  across  both  individual  and  team  goals  is  discretionary,  but  critical 
to  team  performance. 

Deciding  how  to  best  allocate  limited  cognitive  and  behavioral  resources  across  the  multiple  goals  is  a 
fundamental  requirement  that  team  members  must  continuously  evaluate.  Moreover,  making  good  decisions 
about  allocating  limited  resources  is  critically  dependent  upon  monitoring  where  one  stands  with  respect  to 
the  desired  goal  states  and  monitoring  where  one's  teammates  stand.  Investing  limited  resources  toward  the 
achievement  of  individual  goals  may  not  represent  a  good  decision  if  there  are  large  discrepancies  between 
the  team  goal  and  actual  team  performance.  Moreover,  decisions  as  to  which  team  members  ought  to  shift 
resources  to  the  team  goal  will  be  more  effective  to  the  extent  that  individuals  with  the  smallest  individual  goal 
discrepancies  make  the  shift.  Therefore,  dynamic  monitoring  of  multiple  goal  states  and  discrepancies  with 
respect  to  current  performance,  and  making  good  decisions  as  to  team  member  resource  allocation  are 
central  to  effective  team  regulation  and  resource  allocation. 

Theoretical  foundation.  Self-regulation  theory  is  the  dominant  paradigm  for  research  on  the  allocation  of 
attention  and  effort,  and  the  initiation  and  control  of  action.  Although  there  are  several  different  models  of  self¬ 
regulation,  the  models  converge  around  key  features  of  a  process  that  sketches  the  paradigm.  Individuals 
regulate  their  attention  and  effort  (i.e.,  allocate  resources)  around  goals,  monitor  goal  accomplishment  via 
feedback,  and  make  adjustments  in  strategies  and  effort  to  reduce  discrepancies  between  goals  and  current 
performance.  This  approach  has  developed  a  broad  base  of  empirical  support  as  a  general  model  of 
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psychological  processes  that  underlie  human  learning,  motivation,  and  performance  (e  g.,  Karoly,  1993; 
Pintrich,  2003). 

Virtually  all  of  the  research  on  self-regulation  has  focused  on  individuals  striving  to  achieve  single  goals.  Yet, 
working  in  a  team  requires  the  dynamic  regulation  of  individual  cognitive  and  behavioral  resources  with 
respect  to  multiple  goals ,  both  individual  and  team.  This  means  that  the  critical  process  of  how  individuals 
dynamically  allocate  their  resources  around  multiple  goals  has  been  substantially  neglected  in  the  literature. 
Moreover,  the  fact  that  individuals  allocate  attention  and  effort  around  multiple  goals  in  the  team  context 
means  that  regulatory  processes  in  teams  are  multilevel.  Yet,  most  research  targeted  at  improving  team 
performance  either  focuses  only  on  the  individual  level,  ignoring  the  nesting  of  individuals  within  the  team 
context,  or  on  the  team  level  as  a  collective,  ignoring  the  distinctive  contributions  of  individuals  to  team 
processes  and  outcomes.  Our  research  has  treated  team  regulation  and  performance  as  multilevel 
phenomena  that  are  modeled  at  both  levels  simultaneously.  Our  research  program  has  developed  and 
empirically  validated  a  multiple  goal,  multilevel  model  of  individual  and  team  regulation  that  integrates  and 
resolves  these  two  critical  gaps  in  the  literature  (DeShon  et  al.,  2004). 

We  first  developed  a  conceptualization  of  the  influence  of  multiple  goals--individual  and  team--on  feedback 
loops  underlying  the  regulation  of  individual  attention  and  allocation  of  behavioral  resources.  Figure  1 
presents  a  model  of  how  interdependent  feedback  loops  result  in  the  regulation  of  behavior  with  respect  to 
both  individual  and  team  goals.  In  this  model,  two  feedback  loops  have  distinct  individual  and  team  goals  that 
compete  for  control  of  the  individuals'  behavior.  The  feedback  loop  for  the  individual  goal  monitors  individual- 
level  discrepancies  between  current  performance  and  goal  states  and  activates  behavioral  outputs  needed  to 
reduce  the  discrepancy.  The  team  feedback  loop  operates  similarly  on  the  individuals’  team  goals  to  activate 
behavioral  outputs  needed  to  reduce  team-level  discrepancies.  The  behavioral  output  from  each  of  the 
feedback  loops  affects  the  performance  levels  being  regulated  by  the  other  feedback  loop,  such  that  reducing 
discrepancies  for  one  of  the  feedback  loops  will  often  result  in  increased  discrepancies  on  the  other  feedback 
loop.  Finally,  the  initial  characteristics  of  the  situation  and  subsequent  changes  in  the  situation  may  result  in 
increased  discrepancies  or  increased  salience  of  discrepancies  on  one  or  both  of  the  feedback  loops.  As  a 
result,  initial  aspects  of  the  situation  and  changes  in  the  situation  may  bias  the  control  of  behavior  toward 
reducing  discrepancies  at  either  the  team  or  individual  level. 


Figure  1 .  A  multiple  goal  model  of  self-regulation. 
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Next,  we  extrapolated  the  dynamic  self-regulatory  implications  of  the  multiple  goal  resource  allocation  model 
to  develop  a  multilevel  model  that  captured  regulatory  processes  at  both  the  individual  and  team  levels  shown 
in  Figure  2  below.  The  essential  characteristics  required  to  validate  a  multilevel  model  are  (a)  that  team-level 
constructs,  conceptually  parallel  to  those  at  the  individual  level,  satisfy  statistical  criteria  to  support 
composition  (i.e.,  aggregation)  to  the  team  level,  and  (b)  that  the  linkages  among  parallel  constructs  at  both 
levels  demonstrate  functional  equivalence  via  configural  invariance  (Kozlowski  &  Klein,  2000).  In  an 
experimental  design  that  examined  237  trainees  organized  into  79  teams  of  3,  DeShon  et  al.  (2004)  provided 
empirical  support  for  the  multilevel  model  shown  in  Figure  2  and,  indirectly,  for  the  multiple  goal  model  shown 
in  Figure  1 .  Of  particular  importance,  the  relative  salience  of  either  individual  or  team  goal-feedback  loops 
was  the  primary  factor  driving  team  member  resource  allocations  and,  ultimately,  both  individual  and  team 
performance. 

In  essence,  our  research  demonstrated  that  the  key  regulatory  processes  responsible  for  individual  resource 
allocation,  skill  acquisition,  and  performance  also  substantially  hold  at  the  team  level.  Scientifically  we 
validated  a  homologous  multilevel  model  which,  to  the  best  of  our  knowledge,  has  not  been  accomplished  in 
prior  empirical  work  (DeShon  et  al.,2004). 


Team-Regulatory  Processes 


NOTE:  Constructs  above  dashed  line  represent  team-level  constructs.  Constructs  below  line  represent  individual-\e\ie\  constructs. 

Figure  2.  A  multilevel  model  of  self-  and  team-resource  allocation  and  regulation. 

Research  Objectives  and  Approach 

Objectives.  The  validation  of  this  integrated  model  means  that  key  aspects  of  team  skill  acquisition  and 
performance,  those  that  originate  and  emerge  from  parallel  individual-level  self-regulatory  processes,  can  be 
effectively  modeled  in  a  multiple  goal  research  paradigm  where  goals  reference  individual  and  team  resource 
allocation  (or  any  other  multiple  goals  that  compete  for  resources).  Having  established  a  theoretical 
foundation  for  the  importance  of  individual-level  multiple  goal  resource  allocation  to  team  learning  and 
performance,  the  next  logical  step  is  to  focus  more  precisely  on  the  dynamics  of  the  regulation  and  resource 
allocation  process. 

There  are  two  primary  research  foci  that  have  guided  this  effort.  The  first  focus  is  intended  to  extend  our  prior 
use  of  feedback  to  influence  regulation  by  examining  its  impact  on  dynamic  resource  allocation  and,  in 
particular,  how  feedback  characteristics  in  combination  with  a  meta-cognitive  prime  may  enhance  resource 
allocation  processes  with  effects  on  situation  assessment,  strategy  selection,  and  performance  adaptation.  As 
we  noted  in  the  introduction,  resource  allocation  is  a  potential  mechanism  to  account  for  adaptation  to 
unexpected  environmental  perturbations.  Our  prior  research  did  not  address  performance  adaptation,  thus,  it 
is  a  logical  extension  of  the  research  program.  The  second  focus  is  intended  to  build  a  foundation  for 
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extending  our  research  paradigm  such  that  it  can  better  unpack  the  dynamics  of  the  resource  allocation 
process.  Figure  1  illustrates  the  core  resource  allocation  elements  for  the  individual  and  team  feedback  loops: 
goals  serving  as  references  standards,  effort  allocation  toward  the  respective  loops,  performance  conveyed 
by  feedback,  and  the  comparison  between  the  standard  and  current  performance  that  yields  a  discrepancy 
that  influences  behavioral  choice  influencing  resource  allocation  toward  the  individual  or  the  team  loop. 
Although  our  prior  work  extrapolated  this  heuristic  process  to  posit  the  multilevel  model  that  we  validated,  the 
research  did  not  directly  examine  the  dynamic  interplay  among  these  core  elements  over  time.  We  believe 
that  modeling  these  dynamics  over  time  will  yield  important  insights  that  will  enable  us  to  improve  human 
performance  in  complex  task  domains.  In  particular,  this  extension  of  our  paradigm  enables  more  precise 
modeling  of  the  limits  and  wide  variance  of  human  performance  for  dynamic  resource  allocation.  This  serves 
as  a  basis  for  more  accurate  evaluation  of  experimental  interventions  designed  to  improve  multiple  goal 
regulation  and  performance  adaptation.  These  research  foci  are  illustrated  in  Figure  3  below. 

Research  Foci 


12345678? 


Payoff  for  the  Air  Force:  Development  of  principles,  tools,  and  techniques  designed  to  optimize  dynamic  and  adaptive 
resource  allocation  in  teams  to  yield  maximum  performance  under  shifting  task  contingencies. 

v.  .  -  .  -  - y 

Potential  Applications :  Simulation  design,  feedback  systems,  decision  support,  principles  for  training  design. 


Figure  3.  Dynamic  Resource  Allocation  Research  Foci. 

Approach.  We  first  adapted  our  team  radar-tracking  simulation  that  was  used  in  the  prior  funding  cycle  to 
function  as  a  generalized  assignment  task  (e  g.,  Ross  &  Soland,  1975).  As  shown  in  Figure  4,  TEAMSim  is  a 
PC-based,  radar  tracking  simulation  based  on  a  cognitive  task  analysis  that  can  be  configured  to  emulate 
virtually  any  radar  tracking  task  (e.g.  AWACS;  Kozlowski  &  DeShon,  2003).  The  simulation  uses  scripted 
events  that  unfold  in  real  time,  providing  a  shifting  and  emergent  situation  that  demands  adaptability. 
Individuals  or  three  person  teams  are  seated  at  simulated  radar  consoles  that  present  multiple,  dynamically 
interacting  contacts,  contacts  possess  different  characteristics  and  threat  profiles,  and  exhibit  different 
patterns  of  movement.  Participants  (as  individuals  or  interdependent  teams)  must  make  identification 
decisions  and  then  render  an  overall  decision  for  the  contacts.  In  addition,  complex  task  relations  embedded 
in  the  scenario  design  necessitate  shifts  in  task  priorities  and  strategies,  and  in  coordination  requirements 
among  team  members.  TEAMSim  provides  trainees  with  a  dynamic,  self-contained,  and  completely  novel 
task  environment  that  is  appropriate  for  examination  of  complex  skill  acquisition  and  adaptation. 
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Research  Test  Bed:  TEAMSim 

Team  Event-Based  Adaptive  Multilevel  Simulation 


Kozlowski,  S.  W  J.  A  DeShon,  R.  P.  (2004).  A  psychological  fidelity  approach  to  simulation-based  training:  Theory,  research,  and  pnnciples.  In  E.  Salas,  L 
R.  Elliott,  S.  G  Schflett,  A  M.  D  Coovert  (Eds.),  Scaled  worlds:  Development,  validation,  and  applications  (pp.  75-99).  Burlington,  VT:  Ashgate  Pubishing. 

AWACs  Simulation  based  on  a  Cognitive  Task  Analysis 


Figure  4.  TEAMSim. 

Our  prior  research  configured  TEAMSim  to  emulate  3  person  teams  of  AWACS  Weapons  Directors,  In  that 
configuration,  each  team  member  had  to  dynamically  allocate  resources  across  the  processing  of  individual 
and  team  contacts.  Costs  were  incurred  by  either  choice.  If  the  individuals  focused  on  team  targets  they 
incured  costs  associated  with  failing  to  process  the  individual  targets.  If  team  members  focused  on  individual 
contacts  they  incured  costs  associated  with  failing  to  process  team  contacts.  Performance  could  be 
maximized  only  through  the  efficient  and  balanced  processing  of  both  individual  and  team  contacts.  For  the 
current  research  effort,  we  reconfigured  TEAMSim  to  focus  on  individual  resource  allocation  decisions  that 
occur  in  team  contexts.  Participants  were  told  that  they  were  part  of  a  virtual  team  and  that  they  had  both 
individual  and  team  responsibilities;  see  Figure  5.  By  structuring  the  task  in  this  way,  we  were  able  to  focus 
on  individual  resource  allocation  decisions  that  occur  in  team  contexts  without  the  additional  complexities  of 
group  dynamics. 


Figure  5.  TEAMSim  Display  Showing  Individual  (Yellow  Squares)  and  Team  (Blue  Circles)  Contacts. 
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Research  Program  Summary:  Primary  Studies 

Feedback  Cycle  Time  and  Environmental  Change: 

Effects  on  Situation  Assessment,  Resource  Allocation,  and  Adaptation 

Individuals  in  a  team  work  context  have  to  continuously  adapt  their  behaviors  to  maximize  both  individual  and 
team-level  performance.  As  team  members  strive  for  both  individual  and  team  goals,  they  have  a  limited 
amount  of  resources  that  are  available  at  a  given  time.  Thus,  a  choice  must  be  made  between  two  competing 
demands.  Motivation  and  performance  in  teams,  therefore,  involve  a  multilevel,  multiple-goal  process  of 
individual-  and  team-level  regulation  in  which  team  members  make  decisions  concerning  the  allocation  of 
personal  resources  toward  individual  and  team  goals  (See  Figure  1).  Through  this  multiple  goal  regulatory 
process,  team  members’  resource  allocation  decisions  are  continuously  updated  and  evaluated  in  a  way  that 
maximizes  both  individual  and  team  performance.  Therefore,  successful  adaptation  in  teamwork  involves 
effective  management  of  multiple  goal  self-regulatory  processes  such  as  monitoring  performance 
discrepancies  with  respect  to  multiple  goal  states,  and  making  appropriate  resource  allocation  decisions. 

In  a  self-regulation  model  for  individual  behaviors,  feedback  directs  an  individual's  resource  allocation 
decisions  by  providing  knowledge  of  performance  discrepancies  with  respect  to  current  goal  states  (Carver  & 
Scheier,  1998).  In  that  sense,  resource  allocation  as  a  dynamic  process  is  regarded  as  one  promising  means 
to  enhance  situation  assessment  and  diagnosis,  strategy  adjustment,  and  adaptation  when  a  task 
environment  shifts  unexpectedly.  What  characteristics  of  feedback  aid  effective  resource  allocation?  Although 
there  is  a  considerable  literature  on  performance  feedback  which  concludes  that  feedback  is  essential  to 
learning  and  performance,  meta-analytic  findings  indicate  that  feedback  has  positive,  null,  and  even  negative 
effects  (Kluger  &  DeNisi,  1996).  We  theorize  that  these  differential  effects  for  feedback  are  due  to  how  well 
the  feedback  supports  dynamic  goal  regulation  and  resource  allocation. 

In  training  settings,  feedback  or  knowledge-of-results  (KOR)  is  routinely  provided  after  trainees  have 
interacted  with  a  to-be-learned  task.  Although  this  instructional  practice  fits  with  a  heuristic  model  of  the  self¬ 
regulation  process  (i.e.,  goals  set,  goal  striving  behavior  during  practice,  KOR  and  self-reflection  on 
performance-goal  discrepancies,  strategy  and  effort  revisions,  and  iterate  the  cycle),  it  fails  to  account  for  the 
dynamics  of  regulation  and  resource  allocation.  Indeed,  we  believe  that  this  common  form  of  feedback 
provision  slows  and  delays  the  self-regulatory  process,  impedes  resource  allocation,  and  inhibits  adaptation. 
One  key  inference  from  the  multiple  goal  model  is  that  the  cycle  time  for  feedback  (i.e.,  the  rate  at  which  the 
feedback  loop  updates  goal-performance  discrepancy  information),  should  be  commensurate  with  the  rate  of 
change  in  the  task  environment.  Such  feedback  should  provide  real-time  updates  to  the  regulatory  loop, 
thereby  enhancing  situation  assessment  and  diagnosis,  strategy  adjustment,  and  adaptation  when  a  task 
environment  shifts  unexpectedly  To  the  extent  that  feedback  cycle  time  lags  the  rate  at  which  task  events  - 
and  especially  unexpected  environmental  shifts  -  unfold,  self-regulation  and  goal  accomplishment  will  be 
impeded.  Thus,  we  hypothesize  that  fast  feedback  updating  that  is  commensurate  with  the  task  environment 
will  be  more  effective  in  supporting  self-regulation  and  resource  allocation  relative  to  KOR  provided  at  the  end 
of  a  practice  episode.  On  the  other  hand,  there  is  some  very  limited  evidence  which  suggests  that  continuous 
feedback  may  be  so  salient  that  it  would  interfere  with  regulation  and  would  thereby  inhibit  effective  resource 
allocation  (e  g.,  Chhokar  &  Wallin,  1984).  Thus,  it  will  also  be  important  to  evaluate  the  effects  of  fast 
feedback  updating  on  regulation,  resource  allocation,  and  adaptation  relative  to  a  slower  feedback  cycle  time. 

This  experiment  also  investigated  the  effects  of  environmental  change  on  adaptation,  given  different  rates  of 
feedback  cycle  time  and  its  effects  on  self-regulation  and  resource  allocation.  The  literature  on  environmental 
change  contrasts  two  different  and  ubiquitous  types  of  change:  (a)  gradual  or  incremental  change  that  moves 
off  baseline  for  some  period  of  time  and  then  stabilizes  as  a  new  value  or  set  of  environmental  relations  and 
(b)  abrupt  or  metamorphic  change  that  shifts  discontinuously  from  baseline  to  a  new  value  or  set  of 
environmental  relations  (Tushman  &  Romanelli,  1985).  Both  types  of  change  necessitate  adaptation  to  the 
new  values  or  set  of  relations  or  performance  will  be  impeded.  However,  of  the  two  types  of  change,  abrupt 
shifts  are  likely  to  be  more  challenging  because  there  is  less  time  to  detect  and  diagnose  the  change  in 
environmental  values.  In  contrast,  gradual  shifts  afford  more  opportunity  to  detect  a  change  in  environmental 
values.  Because  the  change  is  not  constant,  diagnosis  may  be  difficult,  but  sufficient  information  to  identify 
the  aspect  of  the  environment  that  is  problematic  may  be  possible  such  that  adaptation  is  enhanced  once 
change  has  concluded. 
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Thus,  we  hypothesized  that  participants  would  be  better  able  to  detect  and  diagnose  the  environmental  shift 
under  the  gradual  change  condition  relative  to  abrupt  change  and,  as  noted  above,  that  feedback  cycle  time 
would  facilitate  this  resource  allocation  adaptation.  Thus,  our  focus  is  on  how  well  participants  diagnosed  the 
environmental  change  via  their  feedback,  adjusted  their  strategy,  and  adapted  their  resource  allocation  and 
performance. 

Method 

Participants.  This  experiment  collected  data  from  281  undergraduate  students  recruited  from  psychology 
classes  at  a  large  mid-western  university.  Participants  received  extra-credit  for  their  participation.  Sixty- 
seven  percent  of  the  sample  was  female,  and  eighty-three  percent  was  Caucasian.  The  majority  of 
participants  in  the  sample  were  between  the  ages  of  18  and  22,  with  a  mean  of  20. 

Procedure.  Participants  provided  information  on  their  ACT  or  GRE  scores  as  measures  of  general  cognitive 
ability  via  an  online  questionnaire  prior  to  scheduling  a  lab  session. 

Upon  arrival  at  the  lab,  participants  received  a  brief  training  (approximately  15  minutes)  on  the  simulation.  To 
perform  the  simulation,  participants  had  to  hook  contacts,  query  information  about  these  contacts,  and  then 
make  decisions  based  on  collected  information.  After  training,  participants  were  given  a  chance  to  review  the 
simulation  manual  for  three  minutes,  and  then  perform  in  a  four  minute,  unscored  practice  trial.  Participants 
then  completed  nine  trials  of  the  task,  each  lasting  four  minutes  and  five  seconds.  Each  trial  was  preceded  by 
study  time  during  which  participants  were  given  time  (three  minutes  for  the  first  three  trials,  one  minute 
thereafter)  to  study  the  task  manual.  Each  trial  was  also  followed  by  feedback.  Participants  were  given 
fifteen  seconds  to  review  their  feedback  for  the  previous  trial.  After  receiving  feedback  participants  also  set 
overall  score  goals  for  the  upcoming  trial.  The  entire  series  of  trials  took  approximately  two  hours  to  complete. 

Simulation  scenarios  were  designed  to  prompt  learning  and  resource  allocation.  The  simulation  required 
participants  to  learn  how  to  hook  contacts,  query  information  about  these  contacts,  and  then  make  decisions 
based  on  the  information  cues.  In  addition,  contacts  were  distinguished  as  individual  and  team  types,  along 
with  individual  contacts  belonging  to  the  virtual  teammate.  Participants  were  instructed  to  only  process  those 
contacts  that  were  their  responsibility  as  an  individual  and  a  member  of  the  team  (i.e.  individual  and  team 
contacts).  This  designation  of  contact  type  was  signified  visually  via  contact  symbols;  individual  contact 
symbols  were  yellow  squares  whereas  team  contacts  were  symbolized  by  blue  circles.  Contacts  appeared  on 
the  display  in  pairs  that  were  distinguished  as  low  and  high  priority  via  one  of  the  information  cues. 
Participants  were  told  that  high  priority  contacts  were  worth  more  points  (but  they  were  not  informed  about  the 
point  values).  The  display  was  also  demarked  by  two  perimeters,  represented  by  concentric  circles. 
Participants  lost  points  when  high  priority  contacts  penetrated  the  perimeters,  and  lost  additional  points  for 
every  second  a  high  priority  contact  remained  unprocessed  inside  the  perimeter. 

Effective  processing  necessitated  that  participants  monitor  their  entire  airspace  by  zooming  the  range  of  their 
display  to  monitor  activity  on  both  perimeters,  query  contact  priority,  and  prevent  high  priority  contacts  from 
penetrating  the  defensive  perimeters.  Note,  however,  that  the  strategies  that  would  yield  effective  processing 
were  not  constant  across  the  experiment.  During  early  scenarios,  low  priority  contacts  were  worth  positive 
point  values  and  so  an  effective  strategy  was  to  process  ALL  contacts  as  quickly  as  possible  without 
allocating  resources  differentially.  However,  after  the  environmental  change  low  priority  contacts  assumed 
negative  point  values.  Thus,  after  the  environmental  change,  differential  resource  allocation  directed  toward 
high  priority  contacts  was  necessary  for  effectiveness. 

Experimental  Design.  The  design  was  a  fully  crossed  3  (Feedback  Type)  by  2  (Environmental  Change)  with 
repeated  measures  across  9  trials.  The  first  3  trials  constituted  the  skill  acquisition  phase,  trials  4  to  6 
constituted  the  change  phase,  and  trials  7  through  9  constituted  the  adaptation  phase. 

Feedback  Cycle  Time  Manipulation.  Task  events  (contact  pop-ups)  occurred  approximately  every  10 
seconds.  Thus,  fast  and  slow  feedback  cycle  times  were  selected  to  bracket  the  rate  of  environmental  events. 
Participants  received  one  of  three  different  types  of  feedback  on  all  contacts:  Fast  Cycle  Time  (feedback 
scores  updated  every  one  second),  Slow  Cycle  Time  (feedback  scores  updated  every  40  seconds),  or  End-of- 
Round  (control:  standard  knowledge  of  results  [KOR]  provided  after  the  trial  had  concluded).  Feedback  was 
displayed  separately  for  team  contacts  and  individual  contacts. 
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Fast  cycle  time  was  expected  to  enhance  situation  assessment  (detection  of  environmental  change),  strategy 
adjustment  (a  shift  in  relative  resource  allocation  away  from  low  priority  contacts  to  more  attention  to 
processing  high  priority  contacts),  and  performance  adaptation  (performance  maintenance  or  reduced 
performance  decrements)  relative  to  end-of-round  and  slow  cycle  time.  However,  it  was  an  open  question  as 
to  whether  rapidly  cycling  feedback  might  be  too  salient,  distracting,  and  disruptive  of  resource  allocation. 
Thus,  the  inclusion  of  the  slow  cycling  condition. 

Environmental  Change  Manipulation.  Environmental  change  was  induced  by  modifying  the  rules  of  the  task, 
specifically  the  points  given  for  correctly  processing  low  priority  contacts.  During  training,  participants  were 
instructed  that  high  priority  contacts  were  worth  more  points  than  low  priority  contacts.  For  the  first  three 
trials,  processing  a  high  priority  contact  resulted  in  a  gain  of  200  points,  while  processing  a  low  priority  contact 
resulted  in  a  gain  of  100  points.  The  value  of  high  priority  contacts  was  not  altered,  but  the  value  of  low 
priority  contacts  was  altered  in  one  of  two  ways.  In  the  abrupt  change  condition,  participants  continued  to 
receive  100  points  for  correctly  processing  low  priority  contacts  through  trials  four,  five,  and  six.  However,  the 
task  rules  changed  abruptly  in  the  seventh  trial  (and  persisted  through  the  eighth  and  ninth  trials)  such  that 
processing  low  priority  contacts  (correct  or  incorrect)  yielded  -100  points.  In  the  gradual  change  condition, 
participants  incrementally  lost  points  for  processing  low  priority  contacts  during  trials  four,  five,  and  six  such 
that  the  point  values  were  50,  0,  and  -50,  respectively.  By  trials  seven,  eight,  and  nine,  participants  received  - 
100  points  (equivalent  to  the  abrupt  change  condition). 

We  expected  that  participants  would  be  better  able  to  detect  and  diagnose  the  environmental  shift  under  the 
gradual  change  condition  relative  to  abrupt  change  and,  as  noted  above,  that  feedback  cycle  time  would 
facilitate  this  resource  allocation  adaptation  in  terms  of  how  well  participants  diagnosed  the  environmental 
change  via  their  feedback,  adjusted  their  strategy,  and  adapted  their  resource  allocation  and  performance. 

Measures 


Cognitive  Ability.  Participants  reported  their  ACT  or  SAT  scores,  which  are  proxy  measures  of  general 
cognitive  ability.  The  cognitive  ability  measure  was  created  by  standardizing  ACT  and  SAT  scores  based  on 
national  norms.  It  was  used  as  a  covariate  in  all  analyses. 

Low  Priority  Contacts  (LPC)  Engaged.  The  number  of  LPC  engaged  was  computed  by  summing  the  number 
of  LPCs  participants  processed  in  each  trial.  For  each  trial,  the  maximum  number  of  LPC  engaged  was  30. 
All  participants  were  awarded  100  points  for  correctly  processing  LPCs  in  the  first  3  trials.  For  participants  in 
the  gradual  change  condition,  correctly  processing  LPCs  was  worth  50,  0,  and  -50  in  trials  4,  5,  and  6, 
respectively.  For  participants  in  the  abrupt  change  condition,  processing  LPCs  in  trials  4,  5,  and  6  remained 
at  100  points.  For  all  participants,  correctly  processing  LPCs  in  the  last  3  trials  was  worth  -100  points.  For 
each  trial  and  all  participants,  incorrectly  processing  LPCs  was  worth  -100  points. 

Total  Priority  Queries  (TPQ).  TPQ  was  computed  by  summing  the  number  of  times  participants  queried  the 
priority  level  of  each  contact. 

Engage  Ratio.  The  engage  ratio  was  computed  by  dividing  the  number  of  LPCs  processed  by  the  number  of 
high  priority  contacts  (HPC)  processed  for  each  trial  (i.e.,  Engage  Ratio  =  LPC/HPC).  A  smaller  engage  ratio 
indicates  a  more  effective  resource  allocation  toward  high  priority  contact  processing. 

Performance.  The  total  score  for  each  trial  was  computed  by  summing  the  points  awarded  for  LPC  and  HPC. 
For  all  participants  and  all  trials,  participants  gained  200  points  for  correctly  processing  HPC  and  lost  200 
points  for  incorrectly  processing  HPC.  The  points  awarded  for  LPC  was  as  described  above. 
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Key  Findings 

Repeated  Measures  Multivariate  Analyses  of  Covariance  Variance  (RM-MANCOVA)  were  used  to  evaluate 
the  hypotheses.  Experimental  factors  included:  feedback  cycle  time  (fast,  slow,  EOR)  and  environmental 
change  (gradual  and  abrupt).  Because  our  interest  is  on  resource  allocation  and  adaptation,  analyses  only 
examined  differences  for  the  last  three  trials,  post-environmental  change.  Key  dependent  variables  (DV) 
included  priority  queries,  low  priority  contacts  (LPC)  engaged,  the  ratio  of  low  priority  contacts  engaged  to 
high  priority  contacts  (HPC)  engaged  and  performance  (individual  and  team  contacts)  score.  The  number  of 
LPC  engaged  is  indicative  of  situation  assessment,  that  is,  whether  participants  have  diagnosed  a  shift  in  the 
environment  that  makes  processing  of  low  priority  contacts  costly.  Priority  queries  is  indicative  of  selecting  the 
appropriate  task  strategy  to  enable  differential  resource  allocation,  that  is,  whether  participants  queried  the 
priority  cue  so  as  to  distinguish  types  of  contacts.  The  ratio  of  LPC/HPC  represents  relative  resource 
allocation  such  that  lower  values  are  indicative  of  more  appropriate  resource  allocations  to  high  priority 
contact  processing.  Findings  for  these  variables  are  discussed  below. 

DV:  Number  of  Low  Priority  Contacts  Engaged  (Last  3  Trials).  There  is  a  main  effect,  F(1,  274)  =  14.42,  p  < 
.01,  of  environmental  change  (EC)  such  that  those  who  experienced  gradual  environmental  change  ( M  = 
14.81,  SE  =  .78)  engaged  fewer  low  priority  contacts  than  those  who  experienced  abrupt  environmental 
change  (M  =  19.01,  SE  =  .78).  This  indicates  that  those  who  experienced  gradual  change  were  able  to 
diagnose  and  adapt  to  the  change  better  than  those  who  experienced  abrupt  change,  as  processing  low 
priority  targets  is  dysfunctional  after  the  change  for  reasons  outlined  previously.  Furthermore,  there  is  a  time 
by  feedback  cycle  time  interaction,  F( 4,  548)  =  2.45,  p  =  .05,  such  that  those  who  receive  fast  feedback 
process  fewer  low  priority  contacts  over  time  than  those  who  receive  only  end  of  trial  feedback,  also 
consistent  with  this  interpretation.  Both  findings  support  our  hypotheses.  The  interaction  is  illustrated  in  Figure 
1.1. 


Feedback  cycle  time  by  time  effect  on  LPCs  (last  3  trials) 
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Figure  1.1 


Final  Performance  Report 


Kozlowski  and  DeShon 
Michigan  State  University 


Dynamic  Resource  Allocation 

13 


DV:  Total  Number  of  Priority  Queries  (Last  3  Trials).  In  order  to  execute  the  appropriate  strategy  of 
processing  fewer  low  priority  contacts,  participants  have  to  be  able  to  distinguish  contacts  that  are  low  and 
high  priority.  This  is  accomplished  by  querying  the  priority  cue  and  then  engaging  the  appropriate  contact.  It 
should  be  noted  again  that  prior  to  the  environmental  change,  the  most  effective  strategy  was  to  engage  all 
contacts  as  quickly  as  possible,  thus  ignoring  priority.  Conversely,  after  the  environmental  change  processing 
low  priority  contacts  became  costly,  and  a  functional  strategy  of  processing  only  high  priority  contacts 
required  querying  priority  as  a  means  of  differentiating  high  and  low  priority  contacts.  Thus,  the  number  of 
priority  queries  is  a  critical  means  of  appropriate  strategy  adaptation  and  execution  post  change. 

There  is  a  main  effect,  F( 1,  274)  =  4.51,  p  =  .04,  of  environmental  change  such  that  those  who  experienced 
gradual  environmental  change  (M  =  15.64,  SE  =  1.47)  queried  more  than  those  who  experienced  abrupt 
environmental  change  (M  =  11.21,  SE  =  1.48).  This  is  also  consistent  with  the  above  findings  regarding 
differential  processing  of  targets  for  those  who  experienced  gradual  environmental  change.  Querying  more 
frequently  permitted  more  appropriate  resource  allocation  and  was  an  effective  strategy  in  the  last  three  trials. 

DV:  Engagement  Ratio  (Last  3  Trials).  In  addition  to  processing  fewer  low  priority  contacts  and  querying 
priority  more  frequently,  having  a  lower  ratio  of  low  priority  contact  engagement  to  high  priority  contact 
engagement  was  indicative  of  effective  resource  allocation.  During  the  last  three  trials,  the  most  effective 
resource  allocation  strategy  was  to  engage  only  high  priority  contacts.  A  lower  ratio  indicates  means  that 
resource  allocation  was  more  focused  on  high  priority  contacts 

There  is  a  main  effect,  F(1,  272)  =  13.77,  p  <  .01,  of  environmental  change  such  that  ratio  of  low  priority 
contacts  engaged  to  high  priority  contacts  engaged  was  smaller  for  those  who  experienced  gradual 
environmental  change  (M  =  1.03,  SE  =  .05)  relative  to  those  who  experienced  abrupt  environmental  change 
(M  =  1.29,  SE  =  .05).  There  is  also  a  time  by  feedback  cycle  time  interaction,  F(4,  544)  =  2.54,  p  =  .04,  such 
that  those  who  received  fast  cycling  feedback  have  a  ratio  that  decreases  over  time — indicating  an 
appropriate  shift  in  resource  allocation — while  those  who  receive  only  end  of  trial  feedback  have  a  stable  ratio 
over  time.  Those  who  receive  slow  cycling  feedback  have  an  increasing  ratio,  indicating  a  non-optimal 
strategy.  Both  findings  support  our  hypotheses.  The  time  by  feedback  interaction  is  shown  in  Figure  1.2. 


Feedback  cycle  time  by  time  effect  on  LPC  engaged  to  HPC  engaged  ratio 
(last  3  trials) 
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Figure  1.2 
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DV:  Performance  (Last  3  Trials).  There  is  a  main  effect,  y  =  -608.92,  f( 829.76)  =  -6.61,  p  <  .01 ,  of  the  ratio  of 
low  priority  engagement  to  high  priority  engagement  controlling  for  cognitive  ability,  time,  and  condition  such 
that  individuals  with  a  lower  ratio  performed  better.  This  shows  that  differential  resource  allocation  across 
these  targets  results  in  differing  patterns  of  performance  scores  such  that  those  participants  who  processed 
more  high  priority  contacts  relative  to  low  priority  contacts  performed  more  effectively  on  the  task.  This  is 
consistent  with  our  hypotheses. 

Discussion 


The  pattern  of  findings  reported  previously  was  consistent  with  our  hypotheses.  Gradual  environmental 
change  enable  trainees  to  better  diagnose  and  adapt,  as  expected.  However,  there  is  little  that  can  be  done  to 
influence  the  nature  of  environmental  change.  Thus,  beyond  confirming  the  differential  difficulties  presented 
by  different  forms  of  environmental  change,  there  are  no  substantial  implications  for  training  or  system  design. 
On  the  other  hand,  feedback  cycle  time,  or  the  rate  at  which  a  system  updates  its  state  relative  to  the 
environment  and  regulatory  goals,  is  a  feature  that  can  be  controlled  via  technology  design.  More  rapid 
feedback  cycle  times  enabled  trainees  to  more  effectively  assess  the  changing  and  changed  environmental 
situation  and  to  diagnose  the  need  to  process  fewer  low  priority  contacts,  an  effect  that  strengthened  over 
time.  Participants  receiving  rapidly  updating  feedback  also  showed  a  more  appropriate  relative  resource 
allocation — again,  an  effect  that  strengthened  over  time.  And,  the  more  appropriate  resource  allocation 
predicted  superior  performance.  In  contrast,  conventional  KOR  or  end-of-round  feedback  was  least  effective. 
In  addition,  although  there  was  some  concern  that  rapidly  updating  feedback  might  be  distracting  or 
overwhelming,  there  was  no  evidence  of  detrimental  effects.  Indeed,  compared  to  slower  feedback  updates, 
fast  feedback  yielded  superior  resource  allocation  (Figure  1 .2),  as  those  receiving  slow  feedback  evidenced 
increasingly  inappropriate  resource  allocation  overtime  following  environmental  change. 

Although  these  findings  provide  good  support  for  the  use  of  rapid  feedback  updates  to  support  self-regulation, 
resource  allocation,  and  adaptation,  there  are  two  primary  limitations  that  are  examined  in  a  subsequent 
experiment.  First,  the  number  of  trials  available  for  adaptation  to  be  tracked  (the  last  3  trials)  was  relatively 
brief.  Thus,  an  extension  of  the  design  was  needed  to  lengthen  the  period  of  time  for  adaptation,  particularly 
for  participants  in  the  abrupt  change  condition.  Second,  training  design  research  has  demonstrated  that  a 
variety  of  interventions  which  prompt  active  monitoring  of  regulatory  processes  can  improve  learning  and 
adaptation.  Thus,  an  extension  was  needed  to  couple  feedback  cycle  time  with  a  manipulation  that  influenced 
trainee  meta-cognition. 
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Meta-Cognition,  Feedback  Cycle  Time,  and  Environmental  Change: 

Effects  on  Self-Regulation,  Resource  Allocation,  and  Adaptation 

Recent  theory  (Bell  &  Kozlowski,  in  press)  and  research  (Bell  &  Kozlowski,  in  press)  has  posited  and 
demonstrated  that  there  are  three  primary  psychological  pathways  -  cognitive,  motivational,  and  affective  - 
for  enhancing  self-regulatory  processes,  learning,  and  performance  adaptation.  Of  these,  the  cognitive 
pathway  is  most  potent.  Among  the  many  interventions  that  may  stimulate  this  pathway,  prompting 
metacognition  is  very  promising  (Kozlowski  et  al.,  2001;  Smith,  Ford,  &  Kozlowski,  1997).  Metacognition  has 
been  described  as  planning,  monitoring,  and  revising  goal  appropriate  behavior  (Brown,  Bransford,  Ferrara,  & 
Campione,  1983).  In  other  words,  metacognition  is  an  active  awareness  of  the  process  of  goal  regulation  as 
represented  in  the  multiple  goal  model  (Figure  1).  For  example,  Meloth  (1990)  found  that  specific  instruction 
on  knowledge  of  cognition  led  to  an  increase  of  participants'  metacognition.  This  increase  was  related  to 
better  strategy  use  and  increased  comprehension  performance.  Students  who  were  taught  metacognitive 
strategies  performed  better  in  class  and  were  better  able  to  transfer  knowledge  when  faced  with  novel 
problems  (Volet,  1991).  Schmitt  and  Ford  (2003)  found  that  when  trainees  engaged  in  more  metacognitive 
activity,  they  demonstrated  better  declarative  knowledge,  increased  training  performance,  and  higher  self- 
efficacy  after  controlling  for  both  ability  and  previous  experience.  Thus,  we  hypothesized  that  participants  who 
received  metacognitive  inductions,  that  is,  who  were  prompted  to  self-monitor  their  behavior  during  practice 
trials,  to  gauge  the  effectiveness  of  their  strategies  from  feedback,  and  to  explore  ways  that  they  could 
improve  their  performance  wwould  engage  in  more  effective  self-regulation,  resource  allocation,  and 
adaptation.  In  addition,  we  hypothesized  that  metacogitive  inductions  would  interact  with  feedback  cycle  time 
such  that  metacognitive  inductions  would  enhance  the  positive  effects  of  more  rapid  feedback  updating 
observed  in  the  prior  experiment. 

Method 

Participants.  This  experiment  collected  data  from  577  undergraduate  students  recruited  from  psychology 
classes  at  a  large  mid-western  university.  Participants  received  extra-credit  for  their  participation.  Sixty-two 
percent  of  the  sample  was  female,  and  seventy-two  percent  was  Caucasian.  The  majority  of  participants  in 
the  sample  were  between  the  ages  of  18  and  22,  with  a  mean  of  20. 

Procedure.  Participants  provided  information  on  their  ACT  or  GRE  scores  as  measures  of  general  cognitive 
ability  via  an  online  questionnaire  prior  to  scheduling  a  lab  session. 

Upon  arrival  at  the  lab,  participants  received  a  brief  training  (approximately  15  minutes)  on  the  simulation.  To 
perform  the  simulation,  participants  had  to  hook  contacts,  query  information  about  these  contacts,  and  then 
make  decisions  based  on  collected  information.  After  training,  participants  were  given  a  chance  to  review  the 
simulation  manual  for  three  minutes,  and  then  perform  in  a  four  minute,  unscored  practice  trial.  Participants 
then  completed  nine  trials  of  the  task,  each  lasting  four  minutes  and  five  seconds.  Each  trial  was  preceded  by 
study  time  during  which  participants  were  given  time  (three  minutes  for  the  first  three  trials,  one  minute 
thereafter)  to  study  the  task  manual.  Each  trial  was  also  followed  by  feedback.  Participants  were  given 
fifteen  seconds  to  review  their  feedback  for  the  previous  trial.  After  receiving  feedback  participants  also  set 
overall  score  goals  for  the  upcoming  trial.  The  entire  series  of  trials  took  approximately  two  hours  to  complete. 

Simulation  scenarios  were  designed  to  prompt  learning  and  resource  allocation.  The  simulation  required 
participants  to  learn  how  to  hook  contacts,  query  information  about  these  contacts,  and  then  make  decisions 
based  on  the  information  cues.  In  addition,  contacts  were  distinguished  as  individual  and  team  types,  along 
with  individual  contacts  belonging  to  the  virtual  teammate.  Participants  were  instructed  to  only  process  those 
contacts  that  were  their  responsibility  as  an  individual  and  a  member  of  the  team  (i.e.  individual  and  team 
contacts).  This  designation  of  contact  type  was  signified  visually  via  contact  symbols;  individual  contact 
symbols  were  yellow  squares  whereas  team  contacts  were  symbolized  by  blue  circles.  Contacts  appeared  on 
the  display  in  pairs  that  were  distinguished  as  low  and  high  priority  via  one  of  the  information  cues. 
Participants  were  told  that  high  priority  contacts  were  worth  more  points  (but  they  were  not  informed  about  the 
point  values).  The  display  was  also  demarked  by  two  perimeters,  represented  by  concentric  circles. 
Participants  lost  points  when  high  priority  contacts  penetrated  the  perimeters,  and  lost  additional  points  for 
every  second  a  high  priority  contact  remained  unprocessed  inside  the  perimeter. 
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Effective  processing  necessitated  that  participants  monitor  their  entire  airspace  by  zooming  the  range  of  their 
display  to  monitor  activity  on  both  perimeters,  query  contact  priority,  and  prevent  high  priority  contacts  from 
penetrating  the  defensive  perimeters.  Note,  however,  that  the  strategies  that  would  yield  effective  processing 
were  not  constant  across  the  experiment.  During  early  scenarios,  low  priority  contacts  were  worth  positive 
point  values  and  so  an  effective  strategy  was  to  process  ALL  contacts  as  quickly  as  possible  without 
allocating  resources  differentially.  However,  after  the  environmental  change  low  priority  contacts  assumed 
negative  point  values.  Thus,  after  the  environmental  change,  differential  resource  allocation  directed  toward 
high  priority  contacts  was  necessary  for  effectiveness. 

Experimental  Design.  The  design  was  a  fully  crossed  2  (Metacog nitive  Prime)  by  2  (Feedback  Type)  by  2 
(Environmental  Change)  with  repeated  measures  across  12  trials.  The  first  3  trials  constituted  the  skill 
acquisition  phase,  trials  4  to  6  constituted  the  change  phase,  and  trials  7  through  12  constituted  the 
adaptation  phase.  Primary  interest  is  on  the  adaptation  phase,  trials  7  to  12. 

Metacognitive  Induction  Manipulation.  Probe  questions  were  developed  to  prime  or  induce  metacognitive 
processing.  The  probes  asked  participants  to  what  extent  they  were  (a)  monitoring  their  performance 
feedback,  (b)  evaluating  the  quality  of  their  strategy,  and  (c)  gauging  the  effectiveness  of  their  resource 
allocation.  The  metacognitive  induction  was  provided  following  the  end  of  a  trial  (post-feedback)  and  before 
participants  prepared  for  a  subsequent  trial. 

The  metacognitive  induction  was  expected  to  enhance  self-regulation,  resource  allocation,  and  adaptation. 

Feedback  Cycle  Time  Manipulation.  Participants  received  one  of  two  different  types  of  feedback  on  all 
contacts:  Fast  Cycle  Time  (feedback  scores  updated  every  one  second)  or  End-of-Round  (control:  standard 
knowledge  of  results  [KOR]  provided  after  the  trial  had  concluded).  Feedback  was  displayed  separately  for 
team  contacts  and  individual  contacts. 

Fast  cycle  time  was  expected  to  enhance  situation  assessment  (detection  of  environmental  change),  strategy 
adjustment  (a  shift  in  relative  resource  allocation  away  from  low  priority  contacts  to  more  attention  to 
processing  high  priority  contacts),  and  performance  adaptation  (performance  maintenance  or  reduced 
performance  decrements)  relative  to  end-of-round  feedback.  In  particular,  fast  feedback  cycle  time  was 
expected  to  be  especially  beneficial  under  fast  feedback  updating. 

Environmental  Change  Manipulation.  Environmental  change  was  induced  by  modifying  the  rules  of  the  task, 
specifically  the  points  given  for  correctly  processing  low  priority  contacts.  During  training,  participants  were 
instructed  that  high  priority  contacts  were  worth  more  points  than  low  priority  contacts.  For  the  first  three 
trials,  processing  a  high  priority  contact  resulted  in  a  gain  of  200  points,  while  processing  a  low  priority  contact 
resulted  in  a  gain  of  100  points.  The  value  of  high  priority  contacts  was  not  altered,  but  the  value  of  low 
priority  contacts  was  altered  in  one  of  two  ways.  In  the  abrupt  change  condition,  participants  continued  to 
receive  100  points  for  correctly  processing  low  priority  contacts  through  trials  four,  five,  and  six.  However,  the 
task  rules  changed  abruptly  in  the  seventh  trial  (and  persisted  through  the  trial  12)  such  that  processing  low 
priority  contacts  (correct  or  incorrect)  yielded  -100  points.  In  the  gradual  change  condition,  participants 
incrementally  lost  points  for  processing  low  priority  contacts  during  trials  four,  five,  and  six  such  that  the  point 
values  were  50,  0,  and  -50,  respectively.  In  trials  seven  to  twelve,  participants  received  -100  points 
(equivalent  to  the  abrupt  change  condition). 

We  expected  that  participants  would  be  better  able  to  detect  and  diagnose  the  environmental  shift  under  the 
gradual  change  condition  relative  to  abrupt  change  and,  as  noted  above,  that  meta-cog  nitive  primes  and 
feedback  cycle  time  would  facilitate  this  resource  allocation  adaptation.  Thus,  our  focus  will  be  on  how  well 
participants  diagnosed  the  environmental  change  via  their  feedback,  adjusted  their  strategy,  and  adapted 
their  resource  allocation  and  performance. 

Measures 

Cognitive  Ability.  Participants  reported  their  ACT  or  SAT  scores,  which  are  proxy  measures  of  general 
cognitive  ability.  The  cognitive  ability  measure  was  created  by  standardizing  ACT  and  SAT  scores  based  on 
national  norms.  It  was  used  as  a  covariate  in  all  analyses. 
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Cognitive  Disengagement.  Participants  indicated  whether  or  not  they  had  stopped  striving  to  perform  the 
simulation.  This  variable  was  dummy  coded  as  1  to  indicate  disengagement  and  0  for  not  disengaging.  It  was 
used  as  a  covariate  in  all  analyses. 

Low  Priority  Contacts  (LPC)  Engaged.  The  number  of  LPC  engaged  was  computed  by  summing  the  number 
of  LPCs  participants  processed  in  each  trial.  For  each  trial,  the  maximum  number  of  LPC  engaged  was  30. 
All  participants  were  awarded  100  points  for  correctly  processing  LPCs  in  the  first  3  trials.  For  participants  in 
the  gradual  change  condition,  correctly  processing  LPCs  was  worth  50,  0,  and  -50  in  trials  4,  5,  and  6, 
respectively.  For  participants  in  the  abrupt  change  condition,  processing  LPCs  in  trials  4,  5,  and  6  remained 
at  100  points.  For  all  participants,  correctly  processing  LPCs  in  the  last  3  trials  was  worth  -100  points.  For 
each  trial  and  all  participants,  incorrectly  processing  LPCs  was  worth  -100  points. 

Total  Priority  Queries  (TPQ).  TPQ  was  computed  by  summing  the  number  of  times  participants  queried  the 
priority  level  of  each  contact. 

Engage  Ratio.  The  engage  ratio  was  computed  by  dividing  the  number  of  LPCs  processed  by  the  number  of 
high  priority  contacts  (HPC)  processed  for  each  trial  (i.e.,  Engage  Ratio  =  LPC/HPC).  A  smaller  engage  ratio 
indicates  a  more  effective  resource  allocation  toward  high  priority  contact  processing. 

Performance.  The  total  score  for  each  trial  was  computed  by  summing  the  points  awarded  for  LPC  and  HPC. 
For  all  participants  and  all  trials,  participants  gained  200  points  for  correctly  processing  HPC  and  lost  200 
points  for  incorrectly  processing  HPC.  The  points  awarded  for  LPC  was  as  described  above. 

Key  Findings 

RM-MANCOVA  was  used  to  evaluate  the  hypotheses.  Experimental  factors  included:  metacognitive  primes 
(yes,  no),  feedback  cycle  time  (fast,  EOR)  and  environmental  change  (gradual,  abrupt).  Because  our  interest 
is  on  resource  allocation  and  adaptation,  analyses  only  examined  differences  for  the  last  six  trials,  post- 
environmental  change.  Key  DVs  included  priority  queries,  low  priority  contacts  engaged,  the  ratio  of  low 
priority  contacts  engaged  to  high  priority  contacts  engaged  and  performance  (individual  and  team  contacts) 
score.  The  number  of  LPCs  engaged  is  indicative  of  situation  assessment,  that  is,  whether  participants  have 
diagnosed  a  shift  in  the  environment  that  makes  processing  of  low  priority  contacts  costly.  Priority  queries  is 
indicative  of  selecting  the  appropriate  task  strategy  to  enable  differential  resource  allocation,  that  is,  whether 
participants  queried  the  priority  cue  so  as  to  distinguish  types  of  contacts.  The  ratio  of  LPC/HPC  represents 
relative  resource  allocation  such  that  lower  values  are  indicative  of  more  appropriate  resource  allocations  to 
high  priority  contact  processing.  Findings  for  these  variables  are  discussed  below. 

DV:  Number  of  Low  Priority  Contacts  Engaged  (Last  6  Trials).  As  outlined  in  the  previous  study,  the 
processing  of  low  priority  contacts  is  indicative  of  functional  and  dysfunctional  strategies  at  different  time 
points  in  the  study.  Initially,  low  priority  contacts  provide  a  small  number  of  positive  points.  After 
environmental  change  these  contacts  provide  only  negative  points,  and  thus  resource  allocation  should  shift 
toward  HPC  processing.  A  number  of  means  of  detecting  situational  awareness,  strategy  shift,  and 
differential  resource  allocation  are  examined. 

Consistent  with  our  hypothesis,  there  is  a  main  effect,  E(1,  509)  =  11.18,  p  <  .01,  for  the  metacognitive 
induction  (Ml)  such  that  those  participants  who  received  the  induction  (M  =  16.35,  SE  =  .47)  had  better 
diagnosed  the  changed  situation  by  engaging  fewer  LPCs  than  those  without  the  induction  (M  =  18.77,  SE  = 
.55).  Also  consistent  with  our  hypotheses,  there  is  a  feedback  cycle  time  (FCT)  by  environmental  change  (EC) 
interaction,  E(1 ,  509)  =  5.65,  p  =  .02,  such  that  on  average  those  participants  who  received  fast  feedback 
under  gradual  environmental  change  engaged  the  fewest  low  priority  contacts.  Thus,  on  average  the  gradual 
change,  fast  feedback  condition  yielded  better  adaptation  via  the  reduced  engagement  of  low  priority  contacts 
following  the  stabilization  of  the  environmental  change.  There  were  no  significant  effects  for  the  abrupt 
change  condition.  The  interaction  is  Figure  2.1. 


Final  Performance  Report 


Kozlowski  and  DeShon 
Michigan  State  University 


Dynamic  Resource  Allocation 

18 


Interaction  effect  of  FCT  by  EC  on  processing  of  LPCs  (last  six  trials) 
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Figure  2.1 

There  is  also  a  time  by  feedback  cycle  time  interaction,  F( 5,  2545)  =  2.53,  p  =  .03,  such  that  participants 
receiving  fast  feedback  engaged  fewer  low  priority  contacts  than  those  who  received  end  of  trial  feedback. 
This  difference  increased  across  the  six  post-change  trials  of  the  experiment,  indicating  that  those  receiving 
fast  FCT  continued  to  improve  their  application  of  the  appropriate  task  strategy  as  shown  in  Figure  2.2. 


FCT  condition  by  time  interaction  on  processing  of  LPCs  (last  six  trials) 
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Figure  2.2 
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There  is  also  a  three  way  interaction,  F( 5,  2545)  =  3.32,  p  =  .01,  of  time  by  feedback  cycle  time  by 
metacognitive  induction  such  that  those  participants  with  no  metacognitive  induction  and  fast  cycling  feedback 
engaged  progressively  fewer  low  priority  contacts  over  time  relative  to  those  given  end  of  trial  feedback;  an 
adaptive  behavior.  Those  participants  who  received  the  metacognitive  induction  showed  an  overall 
advantage  for  fast  cycling  feedback  relative  to  those  receiving  end  of  trial  feedback,  but  the  effect  was 
relatively  constant  over  time.  Figures  2.3  and  2.4  illustrate  these  effects. 


Ml  by  FCT  by  time  interaction  on  number  of  LPCs  processed  (last  six  trials) 
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Figure  2.3 
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Figure  2.4 
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Thus,  the  metacognitive  induction  was  effective  at  facilitating  adaptation  to  change  in  the  environment.  Those 
without  the  metacognitive  induction  and  receiving  end  of  trial  feedback  increasingly  processed  low  priority 
contacts;  a  strategy  that  was  dysfunctional.  Those  without  the  metacognitive  induction  but  receiving  fast 
cycling  feedback  decreased  their  processing  of  LPCs  over  time,  appropriately.  However,  the  metacognitive 
induction  offered  a  general  benefit  for  those  participants  receiving  fast  cycling  feedback,  but  there  were  no 
apparent  changes  over  time.  Conventional  outcome  (end  of  trial)  feedback  and  “natural”  self-regulation  (no 
metacognitive  induction)  yielded  the  least  effective  adaptation. 

There  is  also  a  three-way  interaction,  F(5,  2545)  =  2.52,  p  =  .03,  of  time  by  environmental  change  by 
metacognitive  induction  such  that  for  participants  without  the  metacognitive  induction  there  is  no  meaningful 
difference  in  low  priority  contact  processing  across  types  of  environmental  change  or  time.  However,  for 
those  participants  receiving  the  metacognitive  induction,  there  wsa  an  initial  advantage  under  gradual 
environmental  change  relative  to  abrupt  environmental  change.  This  difference  is  likely  due  to  the  opportunity 
to  use  feedback  diagnostically  during  the  change  phase  under  gradual  change.  This  advantage  diminished 
over  time  as  the  participants  in  each  condition  converged  on  the  number  of  low  priority  contacts  they 
processed.  This  interaction  is  depicted  in  Figures  2.5  and  2.6. 


EC  by  Ml  by  time  interaction  on  number  of  LPCs  processed  (last  six  trials) 
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Figure  2.5 


Final  Performance  Report 


Kozlowski  and  DeShon 
Michigan  State  University 


Dynamic  Resource  Allocation 

21 


Metacognitive  Induction 


Environmental  Change 

- Gradual 

- Abrupt 


Figure  2.6 

Thus,  it  can  be  seen  that  those  participants  given  the  metacognitive  induction  and  experiencing  gradual 
environmental  change  did  the  best  initially  in  terms  of  finding  the  appropriate  strategy  of  reduced  low  priority 
contact  processing.  However,  those  given  the  metacognitive  induction  and  experiencing  abrupt 
environmental  change  eventually  converged  to  the  same  strategy.  Thus,  the  metacognitive  induction  provided 
a  robust  improvement  in  adaptation  regardless  of  the  nature  of  environmental  change. 

DV:  Total  Number  of  Priority  Queries  (Last  6  Trials).  As  discussed  earlier,  in  order  to  execute  the  appropriate 
strategy  of  processing  fewer  low  priority  contacts,  participants  have  to  be  able  to  distinguish  contacts  that  are 
low  and  high  priority.  This  is  accomplished  by  querying  the  priority  cue  and  then  engaging  the  appropriate 
contact.  It  should  be  noted  that  prior  to  the  environmental  change,  the  most  effective  strategy  was  to  engage 
all  contacts  as  quickly  as  possible  and  to  ignore  contact  priority,  whereas  after  the  change  processing  low 
priority  contacts  became  costly.  Thus,  the  number  of  priority  queries  is  a  critical  means  of  appropriate  strategy 
adaptation  and  execution  post  change. 

There  is  a  main  effect,  F(1,  509)  =  3.76,  p  =  .05,  of  environmental  change  on  contact  priority  querying  (CPQ) 
in  the  last  six  trials  such  that  those  experiencing  gradual  change  (M  =  10.56,  SE  =  .84)  queried  contact  priority 
more  than  those  experiencing  abrupt  change  (M  =  8.23,  SE  =  .86).  Another  main  effect,  F(1,  509)  =  7.00,  p  = 
.01,  was  found  for  the  metacognitive  induction  in  the  last  six  trials  such  that  those  given  the  induction  ( M  = 
10.99,  SE  =  .78)  queried  contact  priority  more  than  those  without  it  ( M  =  7.81,  SE  =  .92).  Additionally,  there 
was  a  time  by  feedback  cycle  time  interaction,  F( 5,  2545)  =  3.98,  p  <  .01,  in  the  last  six  trials  such  that  those 
receiving  fast  cycling  feedback  queried  contact  priority  more  than  those  participants  given  end  of  trial 
feedback.  This  difference  manifested  primarily  in  the  last  two  trials  and  is  illustrated  in  Figure  2.7. 
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Feedback  cycle  time  by  time  effect  on  CPQ  (last  six  trials) 
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Figure  2.7 

As  noted  previously,  querying  priority  is  an  appropriate  strategy  for  diagnosing  how  to  appropriately  allocate 
resources;  that  is  choosing  to  engage  high  priority  contacts  and  not  processing  low  priority  ones.  As  shown  in 
the  prior  Figure,  total  contact  priority  querying  indicates  whether  participants  queried  contact  priority  as  a 
means  to  allocate  resources  to  high  priority  contacts.  This  resource  allocation  represents  an  adaptive 
behavioral  pattern.  When  feedback  cycle  time  was  fast,  participants  showed  more  total  contact  priority 
querying,  indicating  that  they  were  better  able  to  diagnose  the  environmental  change  relative  to  those  given 
end  of  trial  feedback,  where  a  marked  decrease  in  priority  querying  overtime  can  be  observed. 

There  is  a  time  by  environmental  change  by  metacognitive  induction  interaction,  F( 5,  2545)  =  4.78,  p  <  .01,  on 
contact  priority  querying  in  the  last  six  trials  such  that  those  participants  experiencing  gradual  environmental 
change  queried  more  contact  priorities  than  those  experiencing  abrupt  environmental  change,  with  the 
metacognitive  induction  shifting  this  effect.  To  expound  on  this  finding,  in  the  absence  of  the  metacognitive 
induction  there  is  no  effect  of  environmental  change  over  time.  However,  when  participants  were  given  the 
metacognitive  induction,  there  was  an  initial  advantage  for  those  participants  experiencing  gradual  change 
relative  to  those  experiencing  abrupt  change.  This  is  likely  due  to  the  opportunity  to  discover  the  change 
during  the  change  phase,  evidenced  by  the  fact  that  those  in  the  abrupt  environmental  change  condition 
converge  to  the  same  levels  of  priority  querying  over  time.  This  finding  is  illustrated  in  Figures  2.8  and  2.9. 
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EC  by  Ml  by  time  interaction  on  CPQ  (last  six  trials) 
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Figure  2.8 
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Figure  2.9 

Thus,  the  metacognitive  induction  is  an  effective  intervention  to  ameliorate  the  negative  effects  of 
environmental  change  on  appropriate  strategy  selection  and  adaptive  behavior;  that  is,  using  contact  priority 
querying  to  appropriately  allocate  resources  to  high  versus  low  priority  contact  engagements. 
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DV:  Ratio  of  Low  Priority  Contacts  Engaged  to  High  Priority  Contacts  Engaged.  In  order  to  get  more  directly 
at  the  differential  resource  allocation  noted  above,  we  created  a  ratio  of  low  to  high  priority  contacts  engaged 
such  that  smaller  ratios  indicate  more  optimal  resource  allocation.  While  participants  should  shift  effort  away 
from  the  processing  of  low  priority  contacts,  they  should  also  be  focusing  this  effort  toward  the  processing  of 
high  priority  contacts. 

There  was  a  main  effect,  F(  1,  498)  =  6.69,  p  =  .01,  for  the  metacognitive  induction  on  the  ratio  of  LPC 
engaged  to  HPC  engaged  such  that  those  participants  who  received  the  metacognitive  induction  (M  =  1.18, 
SE  =  .03)  had  a  smaller  ratio,  indicating  appropriate  resource  allocation  toward  high  priority  contacts  and 
away  from  low  priority  contacts  as  compared  to  those  participants  who  had  not  received  the  induction  (M  = 
1.30,  SE=  .04). 


There  was  also  a  feedback  cycle  time  by  environmental  change  interaction,  F(1,  498)  =  5.96,  p  =  .02,  such 
that  those  given  fast  feedback  and  experiencing  gradual  change  had  the  lowest  ratios,  thus  showing  the 
adaptive  benefits  of  fast  feedback  for  diagnosis  when  change  was  gradual  (relative  to  abrupt).  When 
feedback  cycle  time  was  fast  and  environmental  change  was  gradual  participants  were  able  to  effectively 
diagnose  the  appropriate  strategy  of  allocating  more  resources  to  high  priority  contacts  and  thus  curtail 
resource  allocation  to  low  priority  contacts.  This  further  suggests  that  participants  in  the  remaining  conditions 
failed  to  diagnose  the  appropriate  strategy  of  allocating  more  resources  to  high  priority  contacts  and  away 
from  low  priority  contacts.  This  finding  is  illustrated  in  Figure  2.10. 


Environmental  change  by  feedback  cycle  time  interaction  (over  last  six  trials) 


Environmental  Change 

- Gradual 

- Abrupt 


Figure  2.10 
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There  is  also  an  interaction,  F( 5,  2490)  =  3.43,  p  <  .01,  of  feedback  cycle  time  by  time  on  the  ratio  of  LPC 
engaged  to  HPC  engaged  such  that  those  receiving  fast  cycling  feedback  were  better  able  to  diagnose  an 
appropriate  resource  allocation  strategy  to  process  more  HPC  and  fewer  LPC.  Alternatively,  those  receiving 
only  end  of  trial  feedback  increasingly  followed  a  dysfunctional  resource  allocation  strategy  such  that  they 
processed  more  LPCs  over  time.  This  finding  is  illustrated  in  Figure  2.1 1 . 


FCT  by  time  effect  on  LPC  /HPC  engaged  ratio  (last  six  trials) 
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Figure  2.11 

DV:  Performance.  It  is  important  to  demonstrate  that  appropriate  resource  allocation  predicts  performance. 
For  last  6  trials,  there  is  a  main  effect,  y  =  -935.71,  f(2900.99)  =  -17.38,  p  <  .01,  for  engage  ratio  on 
participant  performance  controlling  for  cognitive  ability,  cognitive  disengagement,  time,  and  condition.  Thus, 
those  participants  who  engaged  in  the  appropriate  resource  allocation  strategy  by  focusing  on  processing 
more  high  priority  contacts  relative  to  low  priority  contacts  in  the  last  six  trials  performed  better. 

Qualitative  Analyses 

In  addition  to  the  quantitative  effects  addressed  previously,  we  also  examined  qualitative  statements  obtained 
from  participants.  After  the  experimental  trials  concluded,  participants  were  asked  whether  they  had  detected 
an  environmental  change  and  the  type  of  resource  allocation  strategy  adaptation  needed  to  cope  with  it. 
Those  open-ended  responses  were  coded  into  two  categories  to  capture  the  qualitative  expression  of 
appropriate  strategy  (QEAS):  those  who  deduced  the  strategy  and  those  who  did  not. 

There  is  an  effect  of  QEAS  by  feedback  cycle  time  on  number  of  low  priority  targets  engaged  (F(5,  2545)  = 
3.92,  p  <  .05)  such  that  for  those  participants  receiving  only  end  of  trial  feedback,  those  who  were  able  to 
express  proper  strategy  processed  fewer  low  priority  contacts  across  the  majority  of  trials  relative  to  those 
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who  failed  to  express  the  correct  strategy.  This  contrasts  to  those  participants  in  the  fast  cycling  feedback 
condition.  Of  these  participants,  a  difference  in  the  number  of  low  priority  contacts  processed  between  those 
who  expressed  the  correct  strategy  and  those  who  didn’t  does  not  appear  until  after  the  environmental 
change  has  occurred.  This  is  shown  in  Figures  2.12  and  2.13. 


QEAS  by  feedback  cycle  time  effect  on  LPC  processing  (all  twelve  trials) 
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Figure  2.12 


Participants  receiving  fast  cycling  feedback 


Figure  2.13 
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Recall  that  the  processing  of  low  priority  contacts  is  only  detrimental  in  the  last  six  trials  of  the  experiment  and 
their  processing  in  the  first  six  trials  (first  three  trials  for  those  with  gradual  change)  is  an  appropriate  strategy. 
Thus,  fast  cycling  feedback  not  only  allowed  participants  to  determine  early  in  the  experiment  that  querying 
was  not  useful  and  better  scores  could  be  achieved  by  processing  everything,  but  it  also  allowed  them  to  use 
feedback  diagnostically  later  and  to  realize  that  they  needed  to  query  in  order  to  increase  scores  after  the 
environmental  change. 

This  effect  is  further  illustrated  in  a  QEAS  by  feedback  cycle  time  interaction  on  contact  priority  querying  (F( 5, 
2545)  =  2.88,  p  <  .01).  This  effect  is  consistent  with  the  above  interaction  such  that  for  those  participants  who 
receive  only  end  of  trial  feedback,  those  who  eventually  express  the  correct  strategy  qualitatively  query  more 
than  those  who  don't  express  the  correct  strategy  throughout  the  experiment  -  even  pre-environmental 
change.  Those  given  fast  cycling  feedback  show  no  differences  across  QEAS  for  the  trials  pre-change,  and 
the  appropriate  disjunction  only  occurs  when  the  environmental  change  on  the  task  forces  them  to  adapt. 
This  finding  is  shown  in  Figures  2.14  and  2.15. 


QEAS  by  feedback  cycle  time  on  contact 
priority  querying  (all  twelve  trials) 
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Participants  receiving  fast  cycling  feedback 


Figure  2.15 
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The  implication  of  this  finding  is  that  those  receiving  fast  cycling  feedback  are  able  to  use  the  feedback 
diagnostically  throughout  the  experiment  to  alter  their  strategy  as  the  environment  changed.  Initially,  all 
participants  queried  the  priority  of  contacts,  as  this  was  stipulated  in  the  instructions.  Those  participants  who 
eventually  uncovered  the  environmental  change  receiving  fast  feedback  altered  their  strategy  and  queried 
less  in  order  to  process  more  targets  (an  optimal  strategy  pre-environmental  change),  and  then  again  altered 
their  strategy  post  change  in  order  to  query  and  distinguish  high  and  low  priority  targets  (an  optimal  strategy 
post-environmental  change).  Those  participants  who  eventually  uncovered  the  environmental  change  and 
received  only  end  of  trial  feedback  did  not  alter  their  behavior  early  in  the  experiment  (to  less  querying  of 
priorities)  and  thus  were  not  performing  at  an  optimal  level  during  this  time. 

Discussion 


The  pattern  of  findings  for  experiment  2  essentially  replicated  the  effects  reported  for  experiment  1 ,  and  also 
extended  the  findings  in  important  and  useful  ways.  With  respect  to  replication,  we  again  showed  that  there 
was  an  adaptive  advantage  under  gradual  versus  abrupt  conditions  of  environmental  change  due  to  the 
greater  opportunity  to  detect  unstable  aspects  of  the  environment  and  to  identify  key  aspects  of  the  situation 
relevant  to  those  changing  conditions.  Although  this  finding  in  and  of  itself  is  not  important  because  the  nature 
of  environmental  change  is  not  malleable  in  the  real  world,  what  is  important  are  its  interactions  with  the 
substantive  manipulations  which  represent  training  design  (metacognitive  intervention)  and  embedded 
decision  supports  (feedback  cycle  time);  that  is,  features  that  can  be  used  to  augment  the  effectiveness  of 
human  operators  in  complex  task  environments.  Fast  feedback  was  again  instrumental  to  adaptation,  and 
particularly  advantaged  participants  in  the  gradual  change  condition. 

With  respect  to  extension,  the  use  of  additional  trials  to  observe  adaptation  and  the  addition  of  the 
metacognitive  intervention  to  stimulate  self-regulation  yielded  several  interesting  findings.  First,  the  additional 
time  to  assess  adaptation  showed  that  those  participants  receiving  fast  cycling  feedback  adapted  better  over 
time  in  terms  of  indications  of  appropriate  diagnosis  of  the  situation,  appropriate  strategy  selection,  and  more 
optimal  resource  allocation.  In  addition,  the  qualitative  analyses  that  examined  differences  between  those 
participants  who  deduced  the  appropriate  strategy  and  those  who  did  not  revealed  that  those  receiving  fast 
feedback  were  able  to  better  model  the  appropriate  strategy  across  the  experiment.  That  is,  they  used  the 
most  effective  strategy  early  on  during  the  skill  acquisition  phase  (which  was  to  process  all  contacts  quickly), 
were  able  to  detect  better  during  the  change  phase,  and  were  then  able  to  adapt  their  strategy  and  resource 
allocations  during  the  adaptation  phase.  In  summary,  fast  cycling  feedback  evidenced  several  findings  that 
make  it  a  promising  intervention  for  further  experimental  evaluation  and,  possibly,  application  development. 

The  metacognitive  intervention  also  evidenced  several  promising  findings.  Those  receiving  the  metacognitive 
induction  showed  advantages  in  situation  assessment  and  appropriate  strategy  selection  that  were  amplified 
under  the  gradual  change  condition.  As  above,  we  believe  that  gradual  change  afforded  greater  opportunities 
to  detect  and  diagnose.  Importantly,  however,  those  receiving  the  metacognitive  induction  under  abrupt 
change  evidenced  convergence  with  those  under  gradual  change  over  time.  This  is  an  important  finding 
because  it  shows  that  the  metacognitive  induction  for  self-regulation  improved  the  ability  of  trainees  to 
diagnose  the  nature  of  change  and  to  better  adapt  even  when  the  nature  of  change  was  metamorphic.  We  did 
not  observe  convergence  for  trainees  who  did  not  receive  the  metacognitive  induction.  Moreover,  the 
metacognitive  induction  provided  an  overall  advantage  to  those  receiving  fast  cycling  feedback  in  terms  of 
appropriate  strategy  use,  suggesting  that  these  two  design  features  combine  well.  In  summary,  consistent 
with  the  multiple  goal  regulation  model,  the  metacognitive  intervention  demonstrated  several  positive  effects 
on  resource  allocation  and  adaptation.  It  represents  a  promising  intervention  for  further  experimental 
evaluation  and,  possibly,  application  development.  One  notable  thing  to  bear  in  mind  is  that  the  intervention  is 
exceptionally  easy  to  implement  and  can  be  deployed  in  a  wide  variety  of  training  or  performance  devices. 
Thus,  its  potential  applicability  and  impact  is  high. 

Finally,  although  the  research  findings  are  quite  promising,  they  also  revealed  a  need  to  advance  the 
paradigm.  In  particular,  we  advanced  our  ability  to  model  the  dynamics  of  adaptation  to  environmental 
change.  However,  the  more  specific  micro  dynamics  (i.e.,  within  person  cycles  of  goals,  effort,  and 
performance)  inherent  in  the  multiple  goal  model  remain  elusive  in  the  approach  taken  in  experiments  1  and 
2.  Accordingly,  the  next  step  in  our  evolving  program  sought  to  extend  our  paradigm  to  enable  more  precise 
modeling  of  regulatory  dynamics. 
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Multiple  Goal  Resource  Allocation  and  Skill  Acquisition: 

Investigating  the  Dynamics  of  Self-Regulation 

The  purpose  of  this  investigation  is  to  better  understand  the  dynamic  process  of  regulating  actions  over 
time  in  response  to  performance  feedback.  Although  literally  hundreds  of  motivation  models  exist  that 
specify  dynamic  regulatory  processes,  there  are  very  few  empirical  investigation  of  process  that  actually 
model  dynamics.  This  investigation  serves  as  a  launching  point  for  the  investigation  of  the  dynamic 
processes  that  are  implied  in  our  regulatory  model  (e.g.,  DeShon  et  al. ,  2004).  Although  our  focus  is  on 
the  empirical  evaluation  of  our  model,  the  results  of  this  investigation  speak  to  virtually  all  models  of 
motivation  and  self-regulation  since  these  models  share  a  common  core  set  of  variables  and  processes 
that  are  detailed  next. 

First,  it  is  widely  agreed  that  individuals  possess  desired  states  of  being  (i.e,  goals).  An  individual's 
current  state  of  being  may  be  congruent  with  the  desired  state  (e  g.,  hunger,  thirst,  breathing,  health). 
Frequently,  however,  individuals  possess  desired  states  that  are  currently  incongruent  with  the 
individual's  actual  state  of  being.  In  other  words,  individuals  frequently  want  more  or  less  of  something 
than  is  represented  in  their  current  state  of  being  such  as  such  as  pain,  school  grades,  salary,  leisure 
time,  a  bigger  or  nicer  residence,  respect,  recognition,  control,  etc  ...  Second,  individuals  invest  effort 
over  time  to  maintain  congruent  goals  or  obtain  incongruent  goals.  However,  the  invested  effort  is  not 
uniform  across  tasks  and  goals.  Individuals  invest  tremendous  effort  to  achieve  certain  goals  while 
investing  relatively  less  effort  in  the  pursuit  of  other  goals.  This  empirical  pattern  of  effort  investments 
suggests  that  goal  states  differ  along  one  or  more  dimensions.  At  a  minimum  is  appears  that  goal  states 
are  readily  differentiated  along  the  two  dimensions  of  importance  and  the  anticipated  effort  required  to 
obtain  the  goal  state.  Learning  models  address  the  development  of  these  belief  states.  The  goal 
importance  and  the  effort  believed  to  be  required  to  achieve  the  goal  state  combine  through  a  currently 
unknown  function  to  yield  the  actual  effort  expended  in  the  pursuit  of  a  particular  goal.  Bandura  (1989) 
cogently  described  this  empirical  pattern  of  human  behavior  as: 

“People  motivate  and  guide  their  actions  through  proactive  control  by  setting  themselves  valued  goals 
that  create  a  state  of  disequilibrium  and  then  mobilizing  their  abilities  and  effort  on  the  basis  of 
anticipatory  estimations  of  what  is  required  to  reach  the  goals.” 

Third,  virtually  all  motivation  models  incorporate  some  form  of  an  evaluative  component  that  makes  it 
possible  for  the  individual  to  evaluate  the  effectiveness  of  the  invested  effort  with  respect  to  the  goal.  Is 
the  new  current  state  closer  to  the  goal,  farther  away  from  the  goal,  or  unchanged?  The  theoretical 
details  of  this  process  vary  widely  (e  g.,  comparators  vs.  efficacy  beliefs)  but  empirically  it  appears  that 
individuals  actively  monitor  their  environment  for  knowledge  of  results  or  feedback  information  that  may 
be  used  to  evaluate  the  effect  of  the  invested  effort.  What  happens  at  this  point  is  the  crucible  for 
explanations  of  behavior. 

Once  the  individual  obtains  information  that  is  indicative  of  the  new  current  state  a  many-degree-of- 
freedom  problem  is  encountered.  How  will  the  individual  respond  to  the  information  about  the  current 
state  after  the  effort  investment?  At  a  minimum,  the  individual  may  revise  the  estimate  of  the  effort 
required  to  achieve  the  goal  upward  or  downward.  The  individual  may  revise  the  goal  upward  or 
downward.  The  individual  may  simultaneously  revise  both  the  goal  and  the  effort  estimate  upward  or 
downward  independently.  The  individual  may  also  decide  to  simply  collect  more  information  through 
continued  effort  expenditures  and  leave  the  goal  and  estimated  effort  unchanged. 

Modeling,  the  Dynamics  of_  Motivation 

The  dynamic  process  of  setting  a  goal,  investing  effort  to  maintain  or  obtain  the  goal,  examining 
performance  feedback  to  evaluate  the  success  of  the  effort  investment,  revising  the  goal  and/or  the 
estimate  of  required  effort  investments  for  subsequent  goal  pursuits  that,  in  turn,  affect  subsequent 
performance  and  feedback  is  widely  discussed  in  the  motivation  literature.  However,  the  current  methods 
used  to  investigate  this  dynamic  process  may  not  adequately  capture  the  expected  dynamics.  Two 
methods  currently  dominate  motivational  investigations.  In  fact,  much  more  effort  has  been  devoted  to 
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developing  models  of  behavior  than  to  the  empirical  investigation  of  actual  behavioral  dynamics.  The 
underlying  thesis  of  our  approach  is  that  it  may  prove  useful  to  lay  aside  theoretical  debates  for  a  short 
time  and  let  the  data  speak  a  bit  more  loudly. 


Vector  autoregressive  models  (VAR)  are  a  very  common  set  of  models  used  in  the  linear  dynamic  systems 
literature  to  capture  system  dynamics.  A  finite  order,  covariance  stationary  VAR  model  may  be 
represented  as, 


A0y,=M- i+L  +APy,-p  +  u, 


(1.1) 


where  y,  =  [v„,K  ,yKl  J  is  a  K-dimensional  vector  of  observable  variables,  the  A,,  i  =  0, 1,K  ,p ,  are  (K  x 

K)  coefficient  matrices  and  u,  =  [w,,,K  ,uKt ]  is  a  white  noise  error  process  such  that  the  u,  are  temporally 

uncorrelated  with  zero  mean  and  positive  definite  covariance  matrix  .  The  modeled  lag  length,  p,  is 

termed  the  order  of  the  VAR  process  and  so  the  general  representation  of  the  model  is  VAR(p).  As  an 
example,  a  trivariate  (K  =  3)  VAR(2)  model  may  be  represented  in  expanded  form  as, 


V 

II 

a0 

21 

,0 


a 


a 


a 


a 


\ 

/ 

y* 

= 

/ 

vvJ 

\ 

a 


a 


a 

u 

a' 

2: 

a 1 


a 


\ 

(v  'l 

S\t-\ 

/ 

y*-i 

+ 

/ 

V 

a 


a 


a 


a 


a 


\ 

/ ,,  \ 

/  \ 

yu-2 

uu 

y 2i~2 

+ 

U2, 

/ 

v>w 

^  ' 

(1.2) 


The  coefficient  matrix,  A0,  reflects  the  instantaneous  relationships  between  the  variables  at  time  f,  the 
coefficient  matrix,  Ah  reflects  the  impact  of  the  variables  in  the  immediately  preceding  time  period  (f-1)  on 
the  current  behavior  of  the  system,  and  similarly,  the  coefficient  matrix,  A2,  reflects  the  impact  of  the 
variables  from  two  time  periods  ago  (f-2)  on  the  current  system  behavior.  In  other  words,  the  current  value 
of  ylf  is  modeled  as  a  function  of  the  current  values  of  y2,  and  y3/,  the  immediately  prior  values  of  y1M  and 
y2M  and  y3n,  and  the  even  more  distal  values  of  y1(.2,  y2t.2,  and  y3t.2  from  two  time  periods  prior  to  the 
current  observations.  The  general  model  also  frequently  includes  deterministic  terms  such  as  intercepts  or 
trend  parameters  but  they  are  not  represented  in  (1 .1)  because  the  current  focus  is  on  the  dynamics  of  the 
process.  Equation  1.1  could  be  viewed  as  the  dynamic  model  applied  to  the  residuals  of  a  simple 
regression  on  each  time  series  that  includes  an  intercept,  trend  term(s),  and  cyclical  terms  (if  relevant). 
This  is  a  very  common  model  in  the  multivariate  time  series  literature  and  Lutkpohl  (2005)  provides  an 
excellent,  detailed  description  of  the  model. 

Unfortunately,  the  VAR(p)  model  in  (1.1)  is  not  identified  and,  just  as  in  the  case  of  structural  equation 
modeling,  certain  restrictions  must  be  placed  on  the  coefficients  to  yield  an  identified  model.  The  most 
common  method  of  imposing  identifying  restrictions  on  the  model  in  (1.1)  is  to  restrict  the  A0  matrix  to  an 
identity  matrix  (i.e.,  A0  =  lK)  and  the  resulting  process  is  referred  to  as  a  reduced  form  VAR(p)  process. 
These  restrictions  are  analogous  to  restricting  the  loading  of  one  indicator  for  each  latent  variable  to  1.0  to 
provide  a  scale  for  the  latent  variable  in  structural  equation  modeling.  These  restrictions  are  so  common 
that  reduced  form  VAR(p)  models  are  typically  just  referred  to  as  the  VAR(p)  model.  One  result  of  these 
restrictions  is  that  any  instantaneous  or  contemporaneous  relationships  among  the  variables  is  shuttled  to 
the  error  terms  yielding  uf  that  are  contemporaneously  correlated  and  a  non-diagonal  .  Another  result  is 

that  the  coefficients  in  the  A, . Ap  matrices  are  a  function  of  both  the  coefficients  at  the  given  lag  and  the 

unestimated  coefficients  in  A0  representing  contemporaneous  relations  among  the  variables.  Separate 
ordinary  least  squares  estimation  of  each  equation  in  (1.1)  is  consistent,  asymptotically  efficient  and 
equivalent  to  the  maximum  likelihood  estimator  (Hamilton,  1994). 

The  reduced  form  model  in  (1.1)  summarizes  the  instantaneous  and  intertemporal  relations  among  the 
variables.  However,  the  dynamic  relations  among  the  variables  is  difficult  to  discern  from  the  estimated 
coefficients  in  the  A  matrices.  Also,  the  contemporaneous  influences  are  represented  in  the  covariance 
matrix  of  the  residuals.  Three  general  methods  are  typically  used  to  simultaneously  reflect  the 
instantaneous  and  lagged  relations  among  the  time  series  being  analyzed:  Impulse  response  functions, 
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variance  decompositions,  and  Granger-causality  estimates.  The  focus  here  will  be  on  impulse  response 
functions. 

Impulse  response  functions  represent  the  marginal  responses  of  the  variables  in  the  system  to  an  impulse, 
perturbation,  or  change  in  one  of  the  other  variables  in  the  system.  The  impulse  responses  are  nonlinear 
functions  of  the  coefficients  in  (1.1)  such  that 

=</>u.h(A0,AvK  Ap),  (1.3) 

where  the  ^  h  represent  the  response  of  variable  i  to  an  impulse  in  variable  j,  h  periods  ago.  Viewing  the 
<j>u  h  as  the  if  element  of  the  matrix  <!>,, ,  the  impulse  responses  may  be  computed  recursively  as 

=  h  =  l,2,K,  (1.4) 

)= i 

where  <t>0  =  IK.  In  general,  the  <?>,,,  may  be  viewed  as  sums  of  products  of  the  coefficients  in  the  A, 

matrices.  The  impulse  responses,  along  with  their  associated  confidence  intervals,  are  typically  depicted 
graphically  across  a  limited  number  of  time  periods  to  interpret  the  dynamics  of  how  a  change  in  one 
variable  impacts  one  of  the  other  variables  over  time. 

Panel  Analysis 

To  this  point  the  description  of  the  VAR(p)  model  has  been  based  on  a  data  structure  where  multiple  time 
series  are  observed  on  a  single  unit.  Longitudinal  data  structures  in  psychology,  however,  invariably 
reflect  time  series  assessed  on  multiple  units  (e.g.,  individuals).  Numerous  generalizations  of  the  VAR(p) 
model  have  been  developed  that  incorporate  various  forms  of  unobserved  unit  or  cross-section 
heterogeneity  into  the  estimation  procedure  (e  g.,  Anderson  &  Hsiao,  1981;  Arellano  &  Bond,  1991;  Holtz- 
Eakin,  Newey  &  Rosen,  1988).  The  Generalized-Method-of  Moments  (GMM)  estimators  developed  by 
Arellano  and  Bond  (1991)  and  Arellano  &  Bover,  (1995)  are  useful  when  the  time  dimension  of  the  panel 
data  is  small  (e  g.,  T  <  30)  but  the  fixed  effects  estimator  for  dynamic  panel  data  (also  known  as  within- 
groups  or  least  squares  dummy  variables  estimation)  is  efficient  and  has  negligible  bias  in  large  T  panels 
(Gaduh,  2002;  Judson  &  Owen,  1999)  and  do  not  suffer  from  weak  instrumentation  problems  (e.g., 
Kruiniger,  2000;  Hahn  et  al.,  2001).  The  fixed  effects  estimator  makes  it  possible  to  model  Cross 
sectional  heterogeneity  due  to  omitted  variables  differ  between  cases  but  are  constant  over  time.  This 
analysis  is  analogous  to  allowing  individuals  to  have  different  intercepts  in  a  longitudinal  multilevel  model. 


Method 

Participants.  This  study  collected  data  from  19  undergraduate  students  recruited  from  psychology  classes 
at  a  large  mid-western  university.  Participants  received  extra-credit  for  their  initial  participation  and  earned 
monetary  compensation  by  continuing  the  study  for  nine  follow-up  sessions.  The  sample  was 
predominantly  Caucasian  (70%),  60%  of  participants  were  female,  and  the  mean  age  was  20  years  old. 

Procedure.  Upon  arrival  at  the  lab,  participants  received  a  brief  training  (approximately  15  minutes)  on  the 
task  -  a  complex  PC-based  simulation  of  a  radar  tracking  task.  After  training,  participants  were  given  a 
chance  to  review  the  task  manual  for  three  minutes,  and  then  perform  in  a  four  minute,  unscored  practice 
trial.  This  practice  trial  only  took  place  during  the  initial  experimental  session,  and  not  in  the  follow-up 
sessions.  Within  each  session,  participants  completed  nine  (9)  trials  of  the  task,  each  lasting  four  minutes 
and  five  seconds.  Each  trial  was  preceded  by  study  during  which  participants  were  given  time  (three 
minutes  for  the  first  three  trials,  one  minute  thereafter)  to  study  the  task  manual.  Participants  had  to  learn 
how  to  hook  contacts,  query  information  about  these  contacts  (such  as  their  priority),  and  then  make 
decisions  based  on  the  information.  Each  trial  was  followed  by  feedback  and  trainees  were  given  fifteen 
seconds  to  review  their  performance  on  the  previous  trial.  After  receiving  feedback  they  set  overall  score 
goals  for  the  upcoming  trial.  This  basic  design  was  repeated  for  nine  (9)  addition  experimental  sessions 
yielding  a  total  of  89  cycles  of  goal  setting,  effort  investment,  and  performance  feedback. 
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Task  scenarios  were  designed  to  prompt  learning  and  resource  allocation.  The  basic  task  required 
participants  to  learn  how  to  hook  contacts,  query  information  about  these  contacts  (such  as  their  priority), 
and  then  make  decisions  based  on  the  information.  Contacts  were  distinguished  as  high  and  low  priority, 
and  participants  were  instructed  to  only  focus  on  high  priority  contacts.  In  addition,  contacts  were 
distinguished  as  individual  and  team  contact  types,  along  with  individual  contacts  belonging  to  the  virtual 
teammate.  Participants  were  instructed  to  only  process  those  contacts  that  were  their  responsibility  as  an 
individual  and  a  member  of  the  team  (i.e.  individual  and  team  contacts).  The  display  was  also  demarked 
by  two  perimeters,  represented  by  concentric  circles.  Participants  lost  points  when  contacts  penetrated 
the  perimeters,  and  lost  points  additional  points  for  every  second  a  contact  remained  unprocessed  inside 
the  perimeter.  Effective  processing  necessitated  that  participants  monitor  their  entire  airspace  by  zooming 
the  range  on  the  display  to  monitor  activity  on  both  perimeters,  query  contact  priority,  and  prevent  high 
priority  contacts  from  penetrating  the  defensive  perimeters. 

Measures.  Goal  setting,  effort,  and  performance  were  the  central  measures  for  this  investigation.  For 
each  trial,  effort  was  computed  as  a  linear  composite  of  a  number  of  behavioral  activities  undertaken  in  the 
task  including  hooking  contacts,  making  priority  or  cue  queries  about  contacts,  processing  contacts,  and 
zooming  the  range  on  the  radar  display.  Goals  were  assessed  by  asking  participants  to  write  down  their 
score  goal  for  the  next  round  after  each  trial.  Specifically,  participants  were  asked:  “What  is  your  goal  for 
the  overall  score  for  the  next  round?”  Performance  was  computed  by  summing  participants’  scores  on  low 
priority  contacts,  high  priority  contacts,  and  perimeter  penetrations.  Participants  gained  200  points  for 
correctly  processing  high  priority  contacts  and  lost  200  points  for  incorrectly  processing  high  priority 
contacts.  If  a  high  priority  contacts  crossed  one  of  the  two  defensive  perimeters  than  participants  lost  100 
points  and  were  deducted  5  points  for  each  second  the  high  priority  contacts  was  inside  the  defensive 
perimeter.  Finally,  participants  lost  400  points  for  processing  low  priority  contacts.  Effort  investments 
result  in  performance  only  to  the  extent  that  correct  decisions  are  made  about  how  to  process  contacts. 

Results 


Figure  3.1  presents  the  multiple  time  series  of  goal,  effort,  and  performance  for  each  of  the  nineteen 
participants  in  the  study.  There  are  a  number  of  interesting  patterns  in  the  data.  First,  participants  162, 
265,  336,  and  385  set  goals  that  are  consistently  lower  than  their  performance.  There  is  no  existing 
explanation  for  this  pattern  in  the  self-regulation  literature.  Second,  participants  237  and  378  do  not 
appear  to  be  setting  goals  in  a  manner  that  is  sensitive  to  their  performance.  Their  goals  are  consistently 
set  substantially  higher  than  their  highest  levels  of  performance  on  the  task.  Third,  many  of  the 
participants  (e  g.,  165,  223,  304,  336,  377,  380)  show  a  calibration  period  where  goals  are  initially  set 
either  too  low  or  too  high  relative  to  actual  performance  and  then  gradually  converge  to  levels  that  are 
consistent  with  the  observed  performance  levels.  Fourth,  some  participants  show  much  greater  variance 
in  goal  setting  over  time  than  do  others  (e.g.,  289,  389).  It  is  unclear  whether  these  participants  are  overly 
sensitive  to  performance  feedback  or  whether  some  other  process  accounts  for  the  heterogeneity? 
Finally,  visual  inspection  of  the  three  time  series  does  not  suggest  a  clear  pattern  of  co-movement. 
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The  results  of  a  panel  VAR  analysis  are  presented  in  the  following  section.  First,  a  panel  unit  root  tests  is 
used  to  evaluate  whether  the  time  series  may  be  modeled  as  a  stationarity  process.  Second,  the  process  of 
determining  the  order  of  the  panel  VAR  is  described.  Third,  the  panel  VAR  analysis  is  performed  and 
reported.  Finally,  panel  impulse  response  functions  are  computed  to  interpret  the  system  dynamics. 

Stationarity.  It  is  well-known  that  the  common  unit  root  tests,  such  as  the  augmented  Dickey-Fuller  and 
Phillips-Perron  tests,  have  low  power  to  reject  the  unit-root  null  hypothesis.  Consequently,  many  stationary 
series  are  analyzed  as  if  they  were  non-stationary.  The  use  of  panel  data  in  the  evaluation  of  stationarity 
substantially  improves  the  power  to  reject  a  false  unit-root  null  hypothesis  (Said  &  Dickey,  1984).  The  Im, 
Pesaran,  and  Shin  (1997)  panel  unit  root  test  was  used  in  this  analysis  because  that  data  are  balanced  and 
this  test  does  not  impose  the  restriction  that  all  panel  members  share  a  common  autocorrelation  coefficient 
and  allows  for  heterogenerous  variances.  The  Im,  Pesaran  and  Shin  (1997)  procedure  addresses  the  low 
power  associated  with  single  series  unit-root  tests  by  averaging  the  test  statistics  across  the  panel  members 
and  assuming  i.i.d.  errors.  Drift  and  a  linear  time  trends  were  included  in  all  evaluations  using  this  test.  The 
standardized  t-bar  statistic  (W)  from  the  Im,  Pesaran  and  Shin  approach  rejected  the  unit-root  null  hypothesis 
for  all  three  series:  goal  (-6.81,  p  <  .01),  effort  (W  =  -9.60,  p  <  .01),  and  performance  (W  =  -13.86;  p  <  .01). 

Order  Selection.  The  order  (i.e.,  number  of  lags)  of  the  panel  vector  autoregressive  model  used  to  analyze 
the  data  was  determined  by  examining  the  lag  length  selected  using  the  Akaike  Information  Criterion  (AIC)  for 
vector  autoregressive  models  computed  on  each  person’s  data  separately.  In  one  case,  the  AIC  identified  a 
lag  of  three.  In  all  other  cases,  the  selected  lag  length  was  either  one  or  two.  Based  on  these  results,  it  was 
concluded  that  the  panel  vector  autoregressive  model  should  be  performed  using  an  order  of  two. 

Panel  Vector  Autoregressive  Results.  As  discussed  above,  the  fixed  effects  estimator  for  dynamic  panel  data 
(e  g.,  Baltagi,  2001)  was  used  to  model  cross  sectional  heterogeneity  due  to  omitted  variables  that  differ 
between  cases  but  are  constant  over  time.  The  model  coefficients  and  equation  R 2  fit  indices  from  the  fixed 
effects,  panel  VAR(2)  analysis  are  presented  in  Table  XX.  Coefficients  in  bold  face  font  are  significantly 
different  from  zero  (a  =  .05)  based  on  robust  standard  errors.  The  coefficients  in  this  table  result  from  fitting 
the  reduced  form  model  and  so  should  be  interpreted  cautiously,  if  at  all. 


Table  3.1.  Panel  VAR(2)  coefficients  and  R2  indices. 


Lag  =  1 
Goal 

Effort 

Perf 

Goal 

0.41 

-0.09 

0.46 

Effort 

-0.01 

0.44 

0.03 

Perf 

-0.01 

0.03 

0.26 

Lag  =  2 
Goal 

Effort 

Perf 

R2 

0.19 

0.18 

-0.17 

0.65 

0.01 

0.27 

0.03 

0.58 

0.00 

0.05 

0.20 

0.40 

Impulse  Response  Analysis.  Impulse  response  functions  were  computed  and  graphed  to  examine  the  system 
dynamics  estimated  in  the  panel  VAR  analysis  using  a  cholesky  transformation  of  the  residual  covariance 
matrix.  Figures  3.2  -  3.4  present  the  impact  of  a  one  standard  deviation  impulse  in  goals,  effort,  and 
performance  respectively  on  the  other  two  variables  in  the  system.  The  solid  line  depicts  the  estimated 
impulse  response  and  the  dashed  lines  depict  95%  confidence  intervals.  Closed-form  confidence  intervals  do 
not  yet  exist  for  panel  VAR  models  and  so  the  confidence  intervals  were  computed  using  the  bootstrap 
method  presented  by  Runkle  (1987).  With  respect  to  an  impulse  in  goal,  both  effort  and  performance  show  a 
significant  response  but  performance  appears  to  respond  more  to  a  change  in  goal  than  effort.  This  pattern  is 
curious  given  that  effort  is  thought  to  be  the  primary  vehicle  for  translating  goals  into  performance  (e  g.,  Locke 
and  Latham,  1990).  Further,  for  both  effort  and  performance,  the  impact  of  the  impulse  in  goal  dies  off  slowly 
suggesting  that  goals  can  have  a  relatively  long  lasting  impact  on  both  effort  and  performance.  An  impulse  in 
effort  results  in  a  strong  and  long-lasting  impact  on  goals.  The  impact  on  performance  is  significant  and  also 
dies  out  slowly  but  the  magnitude  of  the  impact  is  relatively  small  in  comparison  to  the  other  effects  depicted 
in  Figures  3.2  -  3.4.  Finally,  an  impulse  in  performance  shows  a  strong  impact  on  subsequent  effort  and 
somewhat  lower,  but  significant,  impact  on  subsequent  goals.  In  all,  the  system  dynamics  have  the 
characteristics  of  a  simultaneous  feedback  system  where  all  variables  impact  subsequent  levels  of  all  other 
variables. 
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Goal  =>  Effort 


Goal  =>  Performance 


Response  to  1  SO  Impulse 


Response  to  1  SD  Impulse 


Tim*  hbnaan 
Shock  in  Y1  on  Y2 


Tim*  Honan 
Shock  in  VI  on  V3 


Figure  3.2.  Response  to  a  one  standard  deviation  impulse  in  Goal. 


Effort  Goal  Effort  =>  Performance 


Figure  3.3.  Response  to  a  one  standard  deviation  impulse  in  Effort. 


Response  to  1  SD  Impulse 


Time  Honzon 
Shock  inY2on  Y1 


Response  to  1  SD  Impulse 


Time  Honan 
Shock  mV2onY3 
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Performance  =>  Effort 


Performance  =>  Goal 


Response  to  1  SD  Impulse 


Response  to  1  SD  Impulse 


Time  Horizon 
Shock  in  Y3  on  VI 
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Time  Honzon 
Shock  in  V3  on  Y2 


Figure  3.4.  Response  to  a  one  standard  deviation  impulse  in  Performance. 


Summary  and  Conclusions 

The  results  presented  above  are  consistent  with  the  dynamics  implied  by  a  classic  feedback  process.  It 
appears  that  changes  in  goals  result  in  immediate  and  lasting  impacts  on  effort  and  performance.  Similarly, 
effort  impacts  goals  and  performance  and  performance  impacts  goals  and  effort  over  time.  In  other  words, 
each  of  these  variables  in  the  self-regulatory  system  dynamically  interact  to  yield  the  system’s  (i.e. , 
individual’s)  behavior.  To  the  best  of  our  knowledge  this  is  the  first  attempt  to  use  modern  stochastic  methods 
to  model  the  dynamics  of  human  regulation  at  the  level  of  individual  behavior. 

Clearly,  this  initial  modeling  attempt  has  numerous  and  substantial  limitations.  However,  two  limitations  stand 
out.  First,  these  results  do  not  demonstrate  causality  and  unmodeled  third  variables  may  actually  be 
responsible  for  the  observed  patterns  of  co-movement  among  the  respective  time  series.  However,  the 
patterns  are  consistent  with  a  very  large  theoretical  literature  and  provide  a  first,  promising  view  of  the 
dynamics  underlying  the  regulation  of  behavior.  Second,  the  current  approach  allowed  for  time-invariant, 
unobserved  heterogeneity.  However,  the  underlying  dynamics  of  the  process  (i.e.,  slopes)  were  assumed  to 
be  homogeneous  across  individuals.  This  assumption  is  likely  unwarranted  and  further  modeling  efforts  are 
required  to  investigate  this  issue. 
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Overall  Conclusions 

The  research  in  this  series  of  projects  examining  the  implications  of  multiple  goal  regulation  on  self-regulation, 
resource  allocation,  and  adaptation  has  once  again  provided  several  promising  findings  that  can  inform  the 
development  of  practical  applications  designed  to  improve  performance  on  complex  tasks  (i.e. ,  simulation 
design,  training  support,  decision  aids)  and,  perhaps  more  importantly,  extensions  to  the  dynamic  goal 
regulation  research  paradigm  we  have  evolved  and  theoretical  developments  that  make  better  use  of  our 
ability  to  model  the  dynamics  of  multiple  goal  resource  allocation  processes  with  greater  precision. 

With  respect  to  practical  applications,  the  findings  from  experiments  1  and  2  confirm  that  rapidly  cycling 
feedback  on  the  state  of  the  environment  relative  goal-performance  discrepancies  was  instrumental  to 
participant  situation  assessment,  strategy  selection,  and  performance  adaptation  following  changes  in  the 
environment  when  compared  to  conventional  KOR  (or  end  of  trial  feedback)  or  dynamic  real-time  feedback 
that  updated  more  slowly.  Although  there  was  some  potential  concern  that  rapidly  cycling  feedback  could  be 
too  salient,  thereby  drawing  on  needed  regulatory  resources,  there  was  no  evidence  for  detriments.  Thus,  one 
would  conclude  that  feedback  needed  to  support  resource  allocation  needs  to  be  provided  at  a  rate  that  is 
commensurate  with  the  cycle  time  necessary  for  goal-performance  discrepancy  updates  to  synchronize  with 
rates  of  change  in  the  performance  environment.  This  principle  can  be  used  to  guide  simulation  design, 
training  support,  and  embedded  decision  aids. 

In  addition  to  replicating  the  primary  findings  of  experiment  1 ,  experiment  2  also  highlighted  the  effectiveness 
of  inducing  metacognitive  processes  as  a  means  to  enhance  trainee  self-regulation,  resource  allocation,  and 
adaptation.  Self-regulation  as  a  psychological  process  is  often  beneath  the  level  of  conscious  awareness. 
Metacognition  is  the  conscious  awareness  and  active  management  of  one's  cognition,  behavior,  and  affect  as 
one  strives  for  goal  accomplishment  and,  therefore,  a  means  to  make  self-regulatory  processes  salient.  We 
hypothesized  that  inducing  active  monitoring  of  one’s  goals,  current  states  (i.e.,  feedback  indicated  goal- 
performance  discrepancies),  and  future  states  (i.e.,  adjustments  of  effort  and  strategies)  would  enhance 
resource  allocation  and  adaptation  and,  moreover,  that  metacognitive  induction  would  supplement  the  prior 
findings  for  feedback  cycle  time.  Those  hypotheses  were  largely  supported.  The  metacognitive  induction 
provided  a  distinct  benefit  to  self-regulation,  resource  allocation,  and  adaptation.  It  can  be  applied  on  its  own 
or  combined  effectively  with  feedback  cycle  time  updates  to  enhance  resource  allocation.  One  practical 
advantage  of  metacognition  induction  is  that  it  can  be  induced  in  a  variety  of  different  ways  including  the 
question  probes  used  in  this  research  embedded  in  technology;  directions,  instructions,  or  training;  or  even  by 
verbal  questions  posed  by  an  instructor.  Thus,  it  is  a  design  principle  with  broad  application  potential. 

With  respect  to  the  evolution  of  our  resource  allocation  research  paradigm  and  theoretical  extension.. 

Experiment  3  represents  an  initial  investigation  of  the  dynamic  processes  underlying  effort  allocation  and 
performance  in  a  simulated  task  environment.  This  research  is  still  in  its  infancy  and  so  practical  applications 
remain  unclear  at  this  time.  However,  with  further  work  along  these  lines  we  hope  to  be  able  to  identify 
important  levers  that  can  be  used  to  push  self-regulatory  dynamics  in  directions  that  yield  high  levels  of 
performance  that  are  maintained  over  time.  The  current  research  adds  to  our  knowledge  of  how  setting  goals 
or  objectives  and  providing  clear  and  immediate  performance  feedback  may  be  used  push  performance  to 
new  plateaus  and  maintain  the  performance  gains.  Future  work  in  this  area  is  needed  to  investigate  the 
dynamic  process  underlying  other  key  levers  including  affective  reactions,  such  as  frustration  and 
dissatisfaction,  and  performance  expectancies. 

We  see  this  research  heading  in  two  complimentary  directions.  First,  as  just  highlighted,  more 
comprehensive  models  of  human  performance  dynamics  need  to  be  developed  and  empirically  evaluated. 
We  have  developed  a  promising  research  methodology  for  investigating  this  issue  and  look  forward  to 
identifying  the  reciprocally  causal  mechanisms  responsible  for  performance  on  complex  tasks  in  both 
individual  and  team  performance  contexts.  Given  the  multivariate  time-series  data  needed  to  explore  these 
issues,  this  research  is  resource  intensive. 

The  second  research  direction  focuses  on  the  exploration  of  the  mechanisms  that  support  and  are 
responsible  for  multiple  goal  regulation,  resource  allocation,  and  adaptation.  Our  intended  approach  to  this 
issue  is  to  use  agent-based  methods  (e.g.,  reinforcement  learning)  to  identify  how  optimal  resource  allocation 
policies  may  be  learned  in  complex  multi-goal  environments  such  as  those  that  exist  in  our  primary  research 
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simulation  task.  Then,  we  intend  to  compare  actual  human  learning  and  performance  on  the  identical  task  to 
identify  aspects  of  human  behavior  and  learning  that  are  responsible  for  sub-optimal  decisions  and 
performance.  Human  learning  and  performance  may  be  improved  by  incorporating  the  reinforcement  learning 
mechanisms  that  frequently  result  in  optimal  performance  into  human  training  and  decision  support 
processes.  In  a  complementary  fashion,  we  believe  that  reinforcememt  learning  approaches  may  be 
improved  by  incorporating  the  known  strengths  that  humans  possess  for  rapid  information  integration,  pattern 
recognition,  resource  allocation,  and  real-time  decision  making.  Together,  such  a  dual  pronged  approach 
would  enable  the  development  of  much  more  robust  and  adaptive  training  and  decision  support  systems  for 
complex  command  and  control  tasks  commonly  used  by  the  US  Air  Force. 
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Research  Program  Summary:  Secondary  Studies 
Enhancing  the  Effectiveness  of  Work  Groups  and  Teams 

Steve  W.  J.  Kozlowski  and  Daniel  R.  Ilgen 

A  monograph  of  this  research  project  was  recently  published: 

Kozlowski,  S.  W.  J.,  &  Ilgen,  D.  R.  (2006).  Enhancing  the  effectiveness  of  work  groups  and  teams. 
Psychological  Science  in  the  Public  Interest,  7,  77-124. 

A  synopsis  of  the  project  was  also  recently  published: 

Kozlowski,  S.  W.  J.,  &  Ilgen,  D.  R.  (2007,  June/July).  The  science  of  team  success.  Scientific  American  Mind, 
54-61 . 


Teams  of  people  working  together  for  a  common  purpose  have  been  a  centerpiece  of  human  social 
organization  ever  since  our  ancient  ancestors  first  banded  together  to  hunt  game,  raise  families,  and  defend 
their  communities.  Human  history  is  largely  a  story  of  people  working  together  in  groups  to  explore,  achieve, 
and  conquer.  Yet,  the  modern  concept  of  work  in  large  organizations  that  developed  in  the  late  19th  and  early 
20th  centuries  is  largely  a  tale  of  work  as  a  collection  of  individual  jobs.  A  variety  of  global  forces  unfolding 
over  the  last  two  decades,  however,  have  pushed  organizations  world-wide  to  restructure  work  around  teams 
to  enable  more  rapid,  flexible,  and  adaptive  responses  to  the  unexpected.  This  shift  in  the  structure  of  work 
has  made  team  effectiveness  a  salient  organizational  concern. 

Teams  touch  our  lives  everyday  and  their  effectiveness  is  important  to  well  being  across  a  wide  range  of 
societal  functions.  There  is  over  50  years  of  psychological  research  -  literally  thousands  of  studies  -  focused 
on  understanding  and  influencing  the  processes  that  underlie  team  effectiveness.  In  this  monograph,  we  sift 
through  this  voluminous  literature  to  identify  what  we  know,  what  we  think  we  know,  and  what  we  need  to 
know  to  improve  the  effectiveness  of  work  groups  and  teams.  We  begin  by  defining  team  effectiveness  and 
establishing  critical  conceptual  considerations  that  underlie  our  approach  to  understanding  it.  We  then  turn  to 
our  review  which  concentrates  primary  attention  on  topics  that  have  established  well-developed  theoretical 
and  empirical  foundations  to  ensure  that  conclusions  and  recommendations  are  on  firm  footing.  As  illustrated 
below,  we  first  focus  attention  on  cognitive,  motivational-affective,  and  behavioral  team  processes  that  enable 
team  members  to  combine  their  resources  to  resolve  team  task  demands  and,  in  so  doing,  achieve 
effectiveness.  Then,  having  established  critical  team  processes  that  enable  team  effectiveness,  we  identify 
interventions  or  levers  that  can  shape  or  align  team  processes  and,  in  that  sense,  provide  tools  and 
applications  for  improving  team  effectiveness.  Topic  specific  conclusions  and  recommendations  are  drawn 
throughout  the  review. 


Figure  1  Conceptual  Framework  and  Review  Focus 


•  Environmental  dynamics  and  complexity  drive  team  task  demands 
•  Team  processes  align  team  member  resources  to  fit  demands 
•  Team  outputs  influence  the  environment 
•  Cycles  are  reciprocal  over  time 


Final  Performance  Report 


Kozlowski  and  DeShon 
Michigan  State  University 


Dynamic  Resource  Allocation 

40 


Team  cognitive  processes  examined  include  team  climate,  team  mental  models  and  transactive  memory,  and 
team  learning.  Team  and  unit  climate  represent  shared  understanding  of  the  key  goals  or  strategic 
imperatives’  driving  the  team  task  environment.  It  creates  the  motivational  press  that  directs  team  member 
resources  and  has  been  linked  to  team  effectiveness.  Team  mental  models  represent  cognitive  structures  that 
organize  important  team  knowledge  areas,  whereas  transactive  memory  represents  team  members' 
knowledge  about  ‘who  knows  what’  that  enables  unique  individual  knowledge  to  be  accessed  by  all  team 
members.  Together,  the  two  concepts  provide  a  means  to  capture  collective  team  knowledge  relevant  to 
performance.  Finally,  team  learning  is  more  of  a  representation  of  the  process  by  which  collective  knowledge 
and  skills  are  acquired.  Looking  across  these  team  cognitive  processes,  team  climate  is  mature  and 
application  ready,  whereas  team  mental  model  research  is  less  well  developed  although  the  concept  is 
application  ready.  Transactive  memory  shows  preliminary  application  potential  and  team  learning  is  still 
undergoing  basic  conceptual  development. 

Team  interpersonal,  motivational,  and  affective  processes  considered  include  cohesion,  efficacy,  and 
potency;  affect,  mood,  and  emotion;  and  conflict.  Among  these  concepts,  cohesion,  efficacy,  and  potency 
have  well  developed  research  foundations  linking  the  processes  to  team  effectiveness.  Team  cohesion  entails 
team  member  attraction,  task  commitment,  and  loyalty.  Team  efficacy  represents  a  shared  confidence  in  the 
team’s  ability  to  accomplish  its  task,  whereas  team  potency  is  a  more  generalized  shared  perception  of  team 
competence.  All  three  team  processes  evidence  the  potential  to  be  influenced,  and  therefore  are  application 
targets.  Research  on  the  other  topics  is  less  well  developed,  so  that  although  they  are  likely  to  be  important 
contributors  to  team  effectiveness,  the  conceptual  and  research  foundations  need  more  elaboration  before 
solid  recommendations  can  be  made. 

The  team  behavioral  processes  that  we  examined  focus  on  team  coordination,  cooperation,  communication; 
enabling  team  member  competencies;  and  the  cognitive-affective-behavioral  processes  by  which  teams 
dynamically  regulate  and  adapt  their  performance.  These  topics  are  a  challenge  to  summarize  succinctly. 
Suffice  it  to  say  that  there  is  a  well  developed  foundation  for  the  person  competencies  that  enable  action  and 
that  underlie  team  coordination  and  performance,  and  there  is  a  confluence  of  promising  work  that  is 
elaborating  the  performance  regulation  and  adaptive  processes  underlying  team  effectiveness.  Several 
techniques  and  approaches  within  this  area  can  be  applied  to  enhance  team  effectiveness  in  specific 
situations. 

Turning  to  the  interventions  or  levers  of  the  team  processes  highlighted  above,  our  review  centered  on  team 
design,  team  training  and  development,  and  team  leadership.  There  is  a  substantial  research  foundation 
supporting  specific  interventions  that  cut  across  these  areas,  although  team  development  is  the  one  area 
where  we  have  lots  of  theory  and  little  solid  data.  Nonetheless,  there  is  considerable  actionable  knowledge  to 
improve  the  design  of  teams  and  their  context,  and  to  use  team  training  to  provide  process  underpinnings  and 
leadership  to  shape  process  development.  Our  recommendations  are  to  apply  the  science  to  enhance  team 
processes  and  team  effectiveness. 

In  sum,  there  is  a  solid  foundation  for  concluding  that  there  is  an  emerging  science  of  team  effectiveness  and 
several  means  to  improve  it.  In  the  concluding  section,  we  summarize  our  primary  findings  to  highlight  specific 
research,  application,  and  policy  recommendations  for  enhancing  the  effectiveness  of  work  groups  and 
teams. 
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A  Motivated  Action  Theory  Account  of  Goal  Orientation 

Richard  P.  DeShon  and  Jennifer  Z  Gillespie 

A  journal  article  reporting  this  research  project  was  published: 

DeShon,  R.  P.  &  Gillespie,  J.  Z.  (2005).  A  motivated  action  theory  account  of  goal  oriented 
behavior.  Journal  of  Applied  Psychology,  90,  1096-  1127. 

Working  in  a  team  imposes  individuals  with  multiple  roles  and  duties  that  are  continuously  updated  as  the 
environment  changes  over  time.  In  order  to  successfully  adapt  to  the  rapidly  changing  environment,  team 
members  often  find  themselves  making  dynamic  decisions  on  where  and  how  to  allocate  their  cognitive  and 
behavioral  resources  across  competing  demands.  These  dynamic  resource  allocations  involve  a  continuous 
process  of  dynamic  monitoring  of  multiple  goal  states  and  discrepancies  with  respect  to  current  performance, 
and  making  good  decisions.  Deciding  how  to  best  allocate  limited  resources  across  multiple  goals  is 
fundamental  for  the  effective  coordination  of  team  members’  behaviors,  which  enable  the  team  to  perform  at 
its  optimal  level. 

One  of  the  essential  motivational  variables  that  predict  resource  allocation  decisions  of  individuals  is 
achievement  motivation.  Achievement  motivation  predicts  how  individuals  allocate  their  resources  toward 
different  requirements  in  achievement  settings,  which  in  turn,  leads  to  learning-  and  performance-related 
behaviors  that  are  critical  for  adaptation  in  changing  environments.  Previous  research  on  achievement 
motivation,  however,  does  not  have  an  adequate  conceptual  framework  for  resource  allocation  decisions  that 
occur  across  multiple  goals  and  over  time.  Working  in  teams  requires  individuals  to  dynamically  shift  their 
goals  over  time  to  meet  the  requirements  of  their  environments.  Knowing  how  different  goals  are  related  and 
how  shifting  between  multiple  goals  occurs  over  time  are  critical  for  advancement  of  team  resource  allocation 
research.  Yet,  achievement  motivation  research  has  an  inconsistent  perspective  on  how  stable  the  goals  are, 
how  many  goals  individuals  pursue,  and  how  multiple  goals  are  determined  by  the  achievement  motivation  of 
individuals.  Based  on  the  extensive  literature  review  in  the  paper,  we  point  out  the  conceptual  and 
methodological  inconsistency  in  the  current  achievement  motivation  literature  that  interfere  with  the 
development  of  integrated  multiple  goal  framework  that  is  essential  for  examining  dynamic  adaptive  behaviors 
of  individuals  in  teams. 

As  illustrated  in  the  following  Figure  5.1,  we  developed  Motivated  Action  Theory  (MAT)  which  encompasses 
numerous  achievement  motivation  perspectives  and  advances  previous  achievement  motivation  studies  by 
imposing  an  integrated  theoretical  framework  for  a  resource  allocation  mechanism  among  multiple  goals. 
Using  the  MAT  framework,  we  argued  that  achievement  motivation  consisted  of  hierarchical  structures  of 
multiple  goals  in  which  high-level  goals  distal  and  desired  status  and  lower  level  goals  are  the  means  to 
achieve  the  higher  level  goals.  Goals  at  different  and  adjunct  levels  are  massively  interconnected  which 
enable  individuals  to  rapidly  shift  their  goals  at  cross-  and  within-levels.  MAT  puts  achievement  motivation  in  a 
dynamic  profile  perspective  in  which  multiple  arrays  of  goals  can  dictate  individuals’  behaviors  over  time. 
Therefore,  over  a  period  of  time,  a  person  may  switch  between  the  various  achievement  goals  and  perform 
sequential  actions  designed  to  reduce  discrepancies  on  more  than  one  higher  level  goal.  Thus,  a  person  can 
flip  back  and  forth  between  a  performance  and  learning  achievement  goal  many  times  over  the  course  of 
working  on  a  task.  Accordingly,  individuals  pursue  multiple  goals  over  time  through  the  dynamics  of 
interaction  between  their  predisposition  in  achievement  motivation  and  the  activation  of  goals  that  are 
imposed  by  changes  in  their  environment. 
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Figure  5.1.  Motivated  Action  Theory  Model  of  the  Goal  Orientation  Hierarchy 
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Locomotion-Assessment,  Action-State  Orientation,  and  Goal  Orientation: 
A  Case  for  Higher  Order  Motives 


Anthony  S.  Boyce,  Goran  Kuljanin,  Guihyun  Park,  Paul  G.  Curran, 
Steve  W.  J.  Kozlowski  and  Richard  P.  DeShon 


Manuscript  Under  Review 


Self-regulatory  process  models  are  the  dominant  approach  to  the  study  of  motivation  and  the  impact  of 
motivation  on  human  cognition  and  behavior.  Although  numerous  models  exist  (e.g.,  Bandura,  1991;  Carver 
&  Scheier,  1990;  Frese  &  Zapf,  1994;  Hacker,  1985;  Kuhl,  1985;  Locke  &  Latham,  1991),  they  share  a  core 
set  of  features  including  goal  choice,  planning  and  effort  allocation  to  achieve  the  chosen  goal,  feedback 
monitoring  to  evaluate  progress  toward  the  goal,  and  affective  and  cognitive  reactions  to  goal  progress 
information.  Further,  virtually  all  self-regulatory  models  assume  that  behavior  is  goal  directed  and  that  goals 
are  hierarchically  structured  such  that  goals  at  high  levels  of  the  hierarchy  specify  why  actions  are  undertaken 
and  lower  level  goals  specify  how  the  higher-level  goals  can  be  met  (e  g.,  Carver  &  Scheier,  1998;  DeShon  & 
Gillespie,  2005,  Powers,  1973). 

The  development  of  self-regulatory  models  initiated  the  pursuit  of  individual  differences  in  the  self-regulatory 
process.  Individual  differences  in  the  self-regulatory  process  address  the  types  of  goals  that  individuals 
choose  to  pursue  (e  g.,  mastery  goals  vs.  achievement  goals  or  approach  vs.  avoid  goals)  and  the  manner  in 
which  the  goals  are  pursued  (e.g.,  planning  vs.  doing,  locomotion  vs.  assessment,  state  vs.  action).  If  goals 
are  hierarchically  structured,  then  the  reason  why  individuals  adopt  different  regulatory  goals  or  methods  for 
achieving  the  goals  may  be  due  to  differences  between  individuals  on  higher  level  goals  (DeShon  &  Gillespie, 
2005). 

This  research  analyzed  two  datasets  that  demonstrated  the  existence  of  two  higher-order  motives  of  task- 
and  ego-involvement  and  their  predictive  relationship  with  self-regulatory  processes  such  as  goal 
commitment,  self-efficacy,  and  cognitive  effort.  After  explaining  theoretical  and  operational  similarities 
between  trait  goal  orientation,  assessment-locomotion,  and  action-state  orientation,  Study  1  concluded  that  a 
second-order  confirmatory  factor  analysis  fit  the  data  as  well  as  a  first-order  confirmatory  factor  analysis.  As 
shown  in  Figure  6.1  below,  mastery  goal  orientation,  locomotion,  and  hesitation  (a  subscale  of  action-state 
orientation)  loaded  highly  on  task-involvement,  whereas  performance  goal  orientation,  performance-avoid 
goal  orientation,  assessment,  and  preoccupation  (a  subscale  of  action-state  orientation)  loaded  highly  on  ego- 
involvement. 


■  First  order  model 

□  Sample  A:  X2=2936(1203),  RMSEA=. 04,  SRMR=. 05 

□  Sample  B:  X2=4648(1 059),  RMSEA=04,  SRMR=.05 

■  Second  order  model 

□  Sample  A:  X2=3336(1216),  RMSEA=. 04,  SRMR=. 08 

□  Sample  B:  X2 -5309(1 072),  RMSEA=.05,  SRMR=.07 


Figure  6.1.  Study  1  Results. 
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Moreover,  as  shown  in  Figure  6.2  below,  Study  2  demonstrated  that  task-involvement  affected  goal 
commitment,  self-efficacy,  and  cognitive  effort  positively  whereas  task-involvement  negatively  affected  off- 
task  thoughts.  On  the  other  hand,  ego-involvement  positively  affected  cognitive  effort  whereas  ego- 
involvement  negatively  affected  self-efficacy  and  affect. 
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Figure  6.2.  Study  2  Results. 

The  current  study  aimed  to  advance  the  self-regulation  literature  by  demonstrating  that  two  higher  order 
motives  of  task-  versus  ego-involvement  encompass  these  three  sets  of  individual  difference  constructs.  From 
a  conceptual  perspective,  this  hierarchical  organization  is  consistent  with  motivated  action  theory  (DeShon  & 
Gillespie,  2005)  which  proposes  that  individual  variability  in  the  selection  of  goals  and  the  means  by  which 
they  are  pursued  is  due  to  differences  on  higher  level  goals.  It  is  also  consistent  with  other  empirical  findings 
that  support  the  presence  of  higher  order  factors  accounting  for  other  lower  level  self-regulatory  constructs 
(e  g.,  Elliot  &  Church,  1997;  Elliot  &  McGregor,  1999;  Heubeck  et  al.,  1998).  Our  results  suggest  that  even 
though  previous  research  has  proposed  that  these  sets  of  individual  difference  constructs  have  unique 
theoretical  origins,  they  do,  in  fact,  share  many  properties  regarding  individual  differences  in  self-regulatory 
processes  that  are  captured  as  parsimonious  higher  order  task-  versus  ego-involvement  motives.  Therefore, 
our  findings  suggest  that  a  gap  exists  in  the  self-regulation  literature  regarding  the  conceptualization  of  unique 
individual  difference  constructs  and  their  empirical  distinctiveness. 

With  respect  to  parsimony,  one  could  argue  that  a  conceptualization  of  higher  order  task-  versus  ego- 
involvement  motives  is  the  foundation  position  against  which  claims  for  the  distinctiveness  of  unique 
constructs  have  to  be  assessed.  However,  we  feel  it  is  somewhat  premature  to  conclude  that  these  individual 
differences  do  not  have  unique  contributions  in  predicting  individual  differences  in  self-regulation.  It  is  not 
clear  whether  the  observed  overlap  among  the  individual  difference  measures  studied  in  this  research  is  due 
to  the  existence  of  two  overarching  motives  that  are  the  common  foundation  for  the  individual  difference 
constructs  or  to  inadequate  operationalization  of  key  theoretical  nuances  in  the  construct  measures,  or  both. 
Future  theory  development  and  empirical  research  needs  to  more  clearly  delineate  the  constructs  to  capture 
their  nuances  and  unique  characteristics  -  if  such  differences  can  be  made  tangible. 

It  is  possible  that  some  of  the  nuanced  distinctions  among  the  construct  sets  could  be  important  and  useful. 
For  example,  goal  orientation  focuses  on  goal  and  evaluative  reactions  to  goal  progress.  Action-state 
orientation  and  locomotion-assessment  are  more  focused  on  the  implementation  of  goal  pursuit  and  its 
maintenance,  but  also  include  evaluative  reaction  to  goal  progress.  For  conceptual  progress,  we  think  a  more 
concerted  effort  is  needed  to  capture  individual  differences  across  a  full  range  of  self-regulatory  phases  that 
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would  either  solidify  a  higher  order  conclusion  or  might  yield  more  empirically  distinctive  self-regulatory 
individual  differences.  Specifically,  self-regulation  can  be  parsed  into  a  more  complete  series  of  phases 
encompassing  goal  choice,  action  initiation,  action  persistence,  feedback  seeking,  and  progress  reactions. 
Next,  potential  individual  differences  in  actions  and  reactions  within  each  phase  can  be  conceptualized.  This 
would  require  conceptual  and  operational  revisions  to  the  current  constructs  and/or  the  construction  of  new 
ones.  If  higher  order  motives  also  capture  variance  across  this  more  conceptually  coherent  set  of  individual 
difference  constructs,  there  would  be  firmer  support  for  the  hierarchical  model.  On  the  other  hand,  if  the 
phases  of  self-regulation  help  to  yield  empirically  distinct  individual  differences,  then  we  would  have  some 
evidence  that  the  higher  order  factors  may  be  due  to  a  failure  to  carefully  parse  distinct  phases  of  self¬ 
regulation  and  to  carelessly  mix  them  together.  Either  outcome  would  constitute  a  conceptual  and  empirical 
advance  in  this  important  area  of  theory  and  research. 
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